Update passive HA application hosts

[ankitgoswami]

Reviewable diff: +1188/-236 across 18 files (excludes generated, test, and story files).

Summary

Adds application-only updates for a passive HA host while preserving the standalone updater path. HA mode is fixed at installation, disables UI-triggered updates, and exposes an explicit local fleet-ha update VERSION workflow that never restarts etcd, Patroni, PostgreSQL, or keepalived. Releases are clean-install-only by default; a reviewed qualification file can authorize one exact source version, and every other source fails before preflight or migration. Normal HA updates require the target GitHub release to be promoted; qualification can authorize only one exact prerelease target. This initial version allows one adjacent transition from a clean-installed substrate and rejects chained application updates.

Stack: #887 -> #888 -> #889 -> #890 -> #891 -> #892. This diff is relative to #889 and relies on #887 for local role checks and #888 for installed paths and services. Interrupted deployment-layout, updater-handoff, and passive-application repair are included here so this PR is safe on its own. Active-host completion and completion-specific reconciliation land in #891; released-artifact qualification lands in #892.

How it works

The release workflow reads one optional stable source tag from qualified-update-from.txt and embeds that tag plus its immutable commit as manifest-covered metadata. The file is empty for the initial clean-install baseline. A stable target intended for adjacent-update qualification names the latest published stable release as its reviewed source before publication and remains a GitHub prerelease until #892 passes; RC releases are clean-install-only in this first version. Before download, the HA updater verifies the target at the fixed official GitHub release API. Normal operation accepts only a promoted release; qualification may authorize one exact prerelease through the root-owned updater environment. Installation fixes the updater mode as standalone or ha. In HA mode, fleet-ha update VERSION requires a current passive node and submits an idempotent request over the updater's local Unix socket. The updater downloads and verifies the release, rejects missing or mismatched source identity, stages and fsyncs it, then the actual stop command rechecks passivity immediately before stopping only fleet-api and fleet-client. Clean installation pins the infrastructure Compose file; application-only updates never advance it, and a missing pin fails closed. Preflight rejects a chained update when the current application has advanced beyond that pinned substrate. Before HA starts after a reboot, a required service preflight reconciles only the interrupted updater handoff and deployment-directory swap. Once the HA substrate is ready, the updater daemon restarts any partially stopped passive application from the current deployment before opening its local API socket. If the handoff restored the previous updater, startup repair revalidates and atomically reinstalls the updater shipped inside the active deployment. The replacement gets one supervised retry while the rollback marker remains until it binds the production socket; repeated HA pre-start repair leaves that daemon-only retry untouched, and a second abrupt daemon failure restores the previous updater instead of creating a restart loop. The updater service is part of the HA unit, so an HA or Docker restart drains it before repair takes the process lock. A busy or failed repair keeps HA stopped and systemd retries; the updater restarts only after HA. After an atomic deployment swap, success requires the requested version, a current passive HA observation, the passive endpoint, and a healthy rolling-update control path. A pre-start activation failure restarts the reconciled deployment immediately. The command tolerates only the expected temporary version mismatch; any other degraded result exits nonzero.

sequenceDiagram
  participant O as Operator
  participant H as fleet-ha
  participant U as Host updater
  participant A as Fleet application
  O->>H: update VERSION
  H->>H: Require healthy passive
  H->>U: Local Unix socket request
  U->>U: Verify source tag and commit, stage, and fsync
  U->>H: Recheck passive and stop Fleet only
  U->>U: Swap deployment
  U->>A: Start and verify VERSION remains passive

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
release workflow, qualification source file, and deployment/version.txt	Embed one explicitly qualified source tag and commit inside verified metadata	Check the fail-closed source and promotion boundary
updater manager and API	Adds fixed HA mode, source enforcement, and the local client protocol	Check standalone remains the default
server/internal/ha/deployment/update.go	Validates passive updates and manages Fleet-only lifecycle	Check strict passive readiness and service boundaries
server/cmd/fleet-ha/	Adds update VERSION	Check operator flow and errors
domain update service	Disables UI-triggered updates in HA mode	Check the first-release product boundary
installer, systemd, and Compose	Pin HA infrastructure outside application releases, repair interrupted layout before HA startup, and grant updater access only to that pinned config directory	Check reboot ordering and that application swaps cannot alter DCS or database services

Key technical decisions & trade-offs

• The target bundle carries one explicitly reviewed source tag and immutable commit; releases do not infer compatibility from tag order.
• An empty qualification file makes the first updater-capable release clean-install-only, so earlier experimental HA installs require reinstall.
• Only stable-to-stable pairs are qualified; RC releases remain clean-install-only.
• A release naming an HA update source stays a prerelease until qualification promotes the unchanged tag and assets; the root-owned qualification override authorizes only one exact target.
• Target migrations must be expand-only and compatible with the running release; Define adjacent-release HA update qualification #892 verifies that contract against released artifacts rather than inferring SQL compatibility at runtime.
• Only Fleet API/client containers are recreated; infrastructure, certificates, and secrets remain untouched.
• The HA-only systemd drop-ins require updater handoff and deployment-layout repair before HA, then start the regular updater after HA; HA restart first stops and drains the updater, while repair failure remains retryable.
• The infrastructure pin is installation state; the updater never reconstructs it from a mutable application release.
• Passivity is checked at admission, inside the actual stop command, and after restart; crash-only takeover handles the remaining check-to-Docker race without a maintenance lease.
• Recovery rechecks local HA state before Compose and leaves a promoted active untouched; automatic and persisted manual recovery both allow either HA role so a takeover is not rejected.
• Pre-start activation failures restart the reconciled application immediately; later failures retain forward recovery because migrations may already have run.
• Existing standalone update behavior is unchanged.

Testing & validation

• Tests cover passive enforcement, missing or wrong source tag/commit metadata, unpromoted-release rejection, exact qualification authorization with representative release metadata, chained-update rejection, staging and swap, failed preflight, active-safe recovery, pre-start application recovery, pre-HA interrupted-layout recovery, bounded updater retry across repeated pre-start repair and rollback, stale updater drop-in rejection, cleanup after updater enablement failure, shared Unix-socket requests, disabled UI updates, and infrastructure preservation.
• Targeted Go tests, Hermit lint, YAML parsing, and the static HA profile check pass.
• Active completion and released-artifact qualification remain in Complete HA updates through bounded failover #891 and Define adjacent-release HA update qualification #892.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (b7277a07d82e12339f5dd06e4bafbdde316a70d9...dc104f7d542db7a490cdc064a7ec53a09e6c9a01, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: NONE

Findings

No concrete security, correctness, or reliability issues were identified in the changed hunks.

Notes

Reviewed only the authoritative PR diff at commit dc104f7d542db7a490cdc064a7ec53a09e6c9a01. No pool, wallet, payout, or stratum-address modifications were introduced.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[Copilot]

Pull request overview

This PR extends the existing host-updater flow to support HA passive-host application-only updates while keeping the standalone updater behavior intact. It introduces a shared Unix-socket updater client, adds an HA deployment mode to the updater manager, and adds a fleet-ha update VERSION operator workflow that updates only fleet-api and fleet-client on a passive HA node.

Changes:

• Add server/internal/updaterapi HTTP-over-unix-socket client and reuse it from the domain updates executor and fleet-ha.
• Add DeploymentMode (standalone vs ha) to the updater manager to run HA-specific preflight/stop/start commands and support rollback to the previous app on failed start.
• Add HA application update helpers (PrepareApplicationUpdate, StopApplication, StartApplication) plus wiring in installer/docs and HA compose to disable UI-triggered updates.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/internal/updaterapi/types.go	Updates package docstring for the shared local updater protocol.
server/internal/updaterapi/client.go	Adds reusable Unix-socket HTTP client with protocol/transport error types.
server/internal/updater/manager.go	Adds HA deployment mode, HA preflight/activation paths, and HA rollback behavior.
server/internal/updater/manager_test.go	Adds HA-mode coverage (passive-only, rollback, preflight failure, interrupted recovery).
server/internal/ha/deployment/update.go	Adds HA application-only update operations (prepare/stop/start + passive verification).
server/internal/ha/deployment/install.go	Installs and configures the host-updater for HA database nodes; exposes ReleaseRoot and compose arg helpers.
server/internal/ha/deployment/install_test.go	Extends golden/witness install assertions for updater installation and enablement.
server/internal/domain/updates/service.go	Maps executor errors to the new updaterapi error types.
server/internal/domain/updates/service_test.go	Updates tests to use updaterapi error types.
server/internal/domain/updates/executor.go	Replaces bespoke unix executor client with updaterapi.Client.
server/internal/domain/updates/executor_test.go	Updates executor tests for updaterapi.Client and error types.
server/cmd/fleet-updater/main.go	Adds --deployment-mode flag/env wiring into updater manager config.
server/cmd/fleet-ha/main.go	Adds fleet-ha update VERSION and internal HA lifecycle subcommands used by the updater.
server/cmd/fleet-ha/main_test.go	Adds tests ensuring passive enforcement and terminal success reporting for update.
deployment-files/ha/tests/test-profile.sh	Adds contract assertions disabling updates in HA and preventing updater socket mounts into services.
deployment-files/ha/README.md	Documents passive-host update workflow (fleet-ha update VERSION).
deployment-files/ha/fleet-compose.yaml	Sets UPDATES_ENABLED=false for HA services (explicit operator-only updates).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf2fe7e12e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5db3353f74

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c363fa3d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dc52fcc5f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ccca78ce1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".