Install HA profile on clean Debian hosts

[ankitgoswami]

Reviewable diff: +1494/-173 across 21 files (excludes generated, test, and story files).

Summary

Adds fleet-ha install for one-command local installation on clean Debian 13 amd64 or arm64 hosts with 4096-byte pages. It validates the host and release, installs the supported packages, persists protected configuration and service ordering, starts only the node's required services, and ends with fleet-ha status --check.

Stack: #887 -> #888 -> #889 -> #890 -> #891 -> #892. This diff is relative to #887 and uses its status command as the final readiness check. Qualification is deferred to #889; application updates are deferred to #890 and #891.

How it works

An operator copies a release, one host-specific node.env, and the matching secrets to each host, then runs fleet-ha install locally. Preflight rejects unsupported platforms, 16K pages, invalid routes or ports, malformed secrets, non-empty HA data directories, stale service overrides, incomplete release artifacts, and a missing HA database image reference before broad package mutation. The offline etcd root password must use the generator's 32-byte lowercase-hex format and differ from every service password and the Fleet JWT signing secret in the host bundle. If the VIP is already present during concurrent installation, ARP must show that it uses the same physical MAC as the configured peer's stable IP; any other owner is rejected. The installer masks Docker and keepalived while packages are installed, applies the firewall first, loads and verifies the exact HA database image named by the packaged Compose file, and prepares the Fleet images. It installs the Docker recovery dependency before etcd first starts, so a Docker or host restart must re-enter the authentication gate. etcd and Patroni use process-failure-only restart policies, so Docker cannot restore them before the HA authentication gate; Docker restart propagation restarts the HA unit after the daemon returns. The installer then starts HA temporarily, proves failover readiness, and enables normal boot persistence. Early snapshot cleanup and later service cleanup use fresh bounded contexts, even after installation is canceled. The first database host rejects principals outside the fixed Fleet allowlist after quorum, clears only an incomplete allowlisted etcd policy, rebuilds its roles, users, and grants from the installed secrets, enables authentication, and verifies all three credentials; peers wait rather than running a second initializer.

flowchart TD
  I["fleet-ha install node.env"] --> P["Validate clean host and release"]
  P --> F["Install packages while services are masked"]
  F --> N["Apply firewall and protected files"]
  N --> E["Start Docker and etcd"]
  E --> R{"Node role"}
  R -->|database| D["Start Patroni, Fleet, keepalived"]
  R -->|witness| W["Keep etcd only"]
  D --> S["fleet-ha status --check"]

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
server/internal/ha/deployment/install.go	Validates and installs a clean host	Check mutation order and failure boundaries
server/internal/ha/deployment/start.go	Starts role-specific services and coordinates etcd auth	Check concurrent bootstrap behavior
server/cmd/fleet-ha/	Adds the local install command	Check file and privilege contracts
HA Compose and systemd files	Persist firewall and service ordering	Check database-host versus witness selection
Dockerfiles and pairing service	Support packaged multi-architecture HA startup	Check release contents and runtime permissions

Key technical decisions & trade-offs

• Installation is local and one-shot; there is no SSH controller, enrollment, resume engine, or rollback framework.
• Privileged work uses sudo; the operator is not added to the Docker group and does not reconnect.
• A 16K-page host is rejected with guidance instead of editing boot configuration or rebooting it.
• Interrupted etcd authentication bootstrap rejects unexpected principals and clears only partial allowlisted state before rebuilding the exact policy, then verifies every expected credential before deleting the offline root secret.
• HA containers restart process failures, while Docker recovery re-enters through systemd so etcd cannot bypass the authentication gate.
• Unexpected Docker loss stops the HA unit and VIP; Docker recovery starts the HA gate again, and failed starts run idempotent container cleanup.
• If the final readiness check times out, installation disables and stops the HA unit while retaining files and staged credentials for diagnosis.
• Host packages come from their signed Debian and Docker repositories; Define clean-install HA qualification #889 binds support to the exact installed versions instead of adding a package snapshot system.
• HA Fleet runs Nmap without raw-network capabilities, preserves documented bounded IPv4 ranges, and skips separate host discovery so configured miner ports remain authoritative.
• Existing standalone installation remains unchanged.

Testing & validation

• Golden install, role selection, cancellation cleanup, service masking, firewall ordering, secret protection, VIP conflict, process-failure restart policy, and 16K-page rejection are covered.
• Targeted Go tests, Hermit lint, and deployment-files/ha/tests/test-profile.sh pass.
• Real clean-host and reboot qualification remains pending in Define clean-install HA qualification #889.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (3392048c7ba154dafaf562d39cab25a4aa22d012...9097b1b8ff7aa00c25f0e934eafbfd431c372996, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: NONE

Findings

No concrete security, correctness, or reliability issues were identified in the authoritative PR diff.

Notes

Review was limited to the specified diff. No pool-address or protobuf changes were present. Registry access was unavailable, so the pinned image digests’ platform coverage could not be independently verified.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[Copilot]

Pull request overview

This PR adds a fleet-ha install workflow to support a one-command local installation of the HA profile onto clean Debian 13 hosts, including role-aware service startup (database vs witness), systemd unit wiring, and updated HA profile artifacts/tests.

Changes:

• Added a host installer (deployment.Install) that validates platform/host inputs, installs required packages, lays down root-owned release/config paths, and performs an initial start/readiness wait.
• Added role-aware service startup/stop entrypoints for systemd (StartInstalledServices / StopInstalledServices) including etcd quorum + auth-bootstrap coordination.
• Updated HA profile artifacts (compose restart policy, systemd ordering, docs/examples) and added installer tests.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
server/internal/ha/deployment/start.go	New role-aware startup/stop logic for installed HA nodes, including etcd auth bootstrap coordination.
server/internal/ha/deployment/preflight.go	Refactors host validation so install can reuse validation without mutating firewall/VIP probing.
server/internal/ha/deployment/install.go	New clean-host installer: platform checks, package installation, file/unit installation, image prep, initial start + readiness wait.
server/internal/ha/deployment/install_test.go	Adds golden-path installer tests (ordering, witness selection, 16K-page rejection).
server/cmd/fleet-ha/main.go	Adds CLI subcommands: install, start, stop, and option parsing.
deployment-files/ha/tests/test-profile.sh	Updates static HA profile assertions (restart policy + unit ordering/exec wiring).
deployment-files/ha/README.md	Updates operator-facing HA profile install flow documentation.
deployment-files/ha/proto-fleet-ha.service	New systemd unit to start HA services with firewall + docker ordering.
deployment-files/ha/proto-fleet-ha-firewall.service	New systemd unit to apply nftables rules before HA services/docker.
deployment-files/ha/node.env.example	Updates example node env guidance for clean-host install flow.
deployment-files/ha/keepalived-systemd.conf.tmpl	Adds unit dependencies so keepalived starts after HA services.
deployment-files/ha/fleet-compose.yaml	Minor compose cleanup in HA fleet overlay.
deployment-files/ha/compose.yaml	Adds restart: always to etcd/Patroni services for resilience.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5a95587d5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19afb509a4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10499c6d12

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32a6e7a048

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a27e256ac5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd2c3ce3c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 57459a4df5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03b0878bd8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".