Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
0507ca7
feat(ha): update passive application hosts
ankitgoswami Aug 7, 2026
3e49abd
Address passive update review feedback (#890)
ankitgoswami Aug 7, 2026
85cd213
Address passive update security review (#890)
ankitgoswami Aug 7, 2026
4e488a6
Surface degraded HA update completion (#890)
ankitgoswami Aug 7, 2026
9c5bc9c
Limit HA updates to adjacent releases (#890)
ankitgoswami Aug 7, 2026
3264788
Keep HA infrastructure pinned during app updates (#890)
ankitgoswami Aug 7, 2026
f0a6617
Make HA recovery command directly runnable (#890)
ankitgoswami Aug 7, 2026
747369f
Verify adjacent releases across bounded history (#890)
ankitgoswami Aug 7, 2026
3056cdd
Preserve HA substrate after target preflight (#890)
ankitgoswami Aug 7, 2026
8d45b9b
Allow verified mixed-version passive startup (#890)
ankitgoswami Aug 7, 2026
4f7199c
Require passive HA update redundancy (#890)
ankitgoswami Aug 7, 2026
ab0c17a
Name rolling-update readiness explicitly (#890)
ankitgoswami Aug 7, 2026
63c3aa1
Harden HA update activation outcomes
ankitgoswami Aug 7, 2026
06cdc9c
Allow passive update after HA role change (#890)
ankitgoswami Aug 8, 2026
ac03249
Fail closed on stale HA update state (#890)
ankitgoswami Aug 8, 2026
952fb4a
Reject stale HA updater artifacts (#890)
ankitgoswami Aug 8, 2026
8749865
Fail closed on incomplete release history
ankitgoswami Aug 8, 2026
76cd355
Document HA updates only with completion flow
ankitgoswami Aug 8, 2026
0baf5f7
Simplify HA update version validation
ankitgoswami Aug 8, 2026
db54679
Report degraded HA update readiness
ankitgoswami Aug 8, 2026
31a8229
Recover updater before HA startup
ankitgoswami Aug 8, 2026
149eb6d
Keep updater independent from HA startup
ankitgoswami Aug 8, 2026
410f722
Backfill pinned HA infrastructure config
ankitgoswami Aug 8, 2026
4315fcc
Clean up HA after updater setup failure
ankitgoswami Aug 8, 2026
421bd22
Disable incomplete HA updater service
ankitgoswami Aug 8, 2026
e6c7393
Permit HA updater config backfill
ankitgoswami Aug 8, 2026
fbd37a8
Keep pinned HA infrastructure immutable
ankitgoswami Aug 8, 2026
9310e8d
Document HA migration compatibility contract
ankitgoswami Aug 8, 2026
614c036
Enforce qualified HA update source
ankitgoswami Aug 8, 2026
e837356
Select adjacent HA release semantically
ankitgoswami Aug 8, 2026
595670d
Require passive role after HA update
ankitgoswami Aug 8, 2026
7f0ff77
Keep stable HA updates on stable channel
ankitgoswami Aug 8, 2026
a6d6b27
Fail HA updates before privileged mutation
ankitgoswami Aug 8, 2026
af71043
Document HA update support boundary
ankitgoswami Aug 8, 2026
7c77493
Require explicit HA update qualification
ankitgoswami Aug 8, 2026
57d1709
Gate HA release publication on qualification
ankitgoswami Aug 8, 2026
478b4a6
Keep HA qualification on released artifacts
ankitgoswami Aug 8, 2026
1b70885
Repair interrupted passive update before HA startup
ankitgoswami Aug 8, 2026
60779bf
Clarify HA update qualification timing
ankitgoswami Aug 8, 2026
175a34b
Recheck passive role during update activation
ankitgoswami Aug 8, 2026
2b444c7
Stabilize concurrent updater admission test
ankitgoswami Aug 8, 2026
8a22eef
Fail closed on unsupported HA update states
ankitgoswami Aug 8, 2026
6f7ea91
Simplify HA update readiness checks
ankitgoswami Aug 8, 2026
a64e003
Bound HA updater recovery and substrate
ankitgoswami Aug 8, 2026
bee09c4
Document initial HA update boundary
ankitgoswami Aug 8, 2026
0a9b8ee
Test HA substrate generation guard
ankitgoswami Aug 8, 2026
6952fd9
Restart updater with HA services
ankitgoswami Aug 8, 2026
648a4c1
Bind HA updates to qualified release builds
ankitgoswami Aug 8, 2026
5e87d28
Backfill pinned HA config before validation
ankitgoswami Aug 8, 2026
0e4223e
Leave a promoted HA node untouched during recovery
ankitgoswami Aug 8, 2026
ca21cdd
Hold HA qualification releases as prereleases
ankitgoswami Aug 8, 2026
75d82a6
Fail closed before replacing a running HA app
ankitgoswami Aug 8, 2026
85e5b31
Allow HA repair when updater is already healthy
ankitgoswami Aug 8, 2026
ac0d8ab
Qualify HA updates from stable releases only
ankitgoswami Aug 8, 2026
3574c44
Recover interrupted updater handoffs
ankitgoswami Aug 8, 2026
7354e2e
Recover passive HA application on updater restart
ankitgoswami Aug 8, 2026
0ddf508
Preserve updater rollback during restart recovery
ankitgoswami Aug 8, 2026
488ce8d
Recover HA applications after substrate startup
ankitgoswami Aug 8, 2026
7bbb6ff
Fail closed without pinned HA infrastructure
ankitgoswami Aug 8, 2026
60271a8
Keep HA rolling updates stable-only
ankitgoswami Aug 8, 2026
9913fd4
Preserve updater retry through HA repair
ankitgoswami Aug 8, 2026
97bee41
Require qualified HA update releases
ankitgoswami Aug 8, 2026
06dd65b
Accept complete GitHub release metadata
ankitgoswami Aug 8, 2026
ac409b5
Make HA recovery commands active-safe
ankitgoswami Aug 8, 2026
f76a510
Align HA updates with local status API
ankitgoswami Aug 10, 2026
25fadf0
Remove duplicate HA update setting
ankitgoswami Aug 11, 2026
11a1d15
Adapt HA update validation to installer simplification (#890)
ankitgoswami Aug 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/proto-fleet-artifact-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ on:
description: Version string used in artifact filenames and embedded metadata.
required: true
type: string
ha_update_from_version:
description: Exact prior release allowed to use this bundle for an HA application update.
required: false
default: ""
type: string
ha_update_from_commit:
description: Commit of the exact prior release allowed to use this bundle for an HA application update.
required: false
default: ""
type: string
build_date:
description: Optional UTC ISO-8601 build date. Defaults to the workflow run time.
required: false
Expand Down Expand Up @@ -39,6 +49,8 @@ jobs:
build_date: ${{ steps.metadata.outputs.build_date }}
channel: ${{ steps.metadata.outputs.channel }}
is_prerelease: ${{ steps.metadata.outputs.is_prerelease }}
ha_update_from_version: ${{ steps.metadata.outputs.ha_update_from_version }}
ha_update_from_commit: ${{ steps.metadata.outputs.ha_update_from_commit }}
version: ${{ steps.metadata.outputs.version }}
steps:
- name: Resolve artifact metadata
Expand All @@ -47,6 +59,8 @@ jobs:
INPUT_BUILD_DATE: ${{ inputs.build_date }}
INPUT_CHANNEL: ${{ inputs.channel }}
INPUT_IS_PRERELEASE: ${{ inputs.is_prerelease }}
INPUT_HA_UPDATE_FROM_VERSION: ${{ inputs.ha_update_from_version }}
INPUT_HA_UPDATE_FROM_COMMIT: ${{ inputs.ha_update_from_commit }}
INPUT_VERSION: ${{ inputs.version }}
run: |
if [[ ! "$INPUT_VERSION" =~ ^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$ ]] || [[ "$INPUT_VERSION" == "latest" ]]; then
Expand All @@ -59,6 +73,17 @@ jobs:
exit 1
fi

if [[ -n "$INPUT_HA_UPDATE_FROM_VERSION" ]] && [[ ! "$INPUT_HA_UPDATE_FROM_VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$ ]]; then
echo "::error::HA update source '$INPUT_HA_UPDATE_FROM_VERSION' must be a stable or RC release tag."
exit 1
fi
if { [[ -n "$INPUT_HA_UPDATE_FROM_VERSION" ]] && [[ -z "$INPUT_HA_UPDATE_FROM_COMMIT" ]]; } || \
{ [[ -z "$INPUT_HA_UPDATE_FROM_VERSION" ]] && [[ -n "$INPUT_HA_UPDATE_FROM_COMMIT" ]]; } || \
{ [[ -n "$INPUT_HA_UPDATE_FROM_COMMIT" ]] && [[ ! "$INPUT_HA_UPDATE_FROM_COMMIT" =~ ^[a-f0-9]{40}$ ]]; }; then
echo "::error::HA update source version and 40-character commit must be provided together."
exit 1
fi

if [[ -n "$INPUT_BUILD_DATE" ]]; then
if [[ ! "$INPUT_BUILD_DATE" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]]; then
echo "::error::build_date '$INPUT_BUILD_DATE' must be an ISO-8601 UTC timestamp (e.g. 2026-04-24T12:00:00Z)."
Expand All @@ -72,6 +97,8 @@ jobs:
echo "build_date=$BUILD_DATE" >> "$GITHUB_OUTPUT"
echo "channel=$INPUT_CHANNEL" >> "$GITHUB_OUTPUT"
echo "is_prerelease=$INPUT_IS_PRERELEASE" >> "$GITHUB_OUTPUT"
echo "ha_update_from_version=$INPUT_HA_UPDATE_FROM_VERSION" >> "$GITHUB_OUTPUT"
echo "ha_update_from_commit=$INPUT_HA_UPDATE_FROM_COMMIT" >> "$GITHUB_OUTPUT"
echo "version=$INPUT_VERSION" >> "$GITHUB_OUTPUT"

{
Expand Down Expand Up @@ -434,6 +461,8 @@ jobs:
env:
BUILD_DATE: ${{ needs.metadata.outputs.build_date }}
CHANNEL: ${{ needs.metadata.outputs.channel }}
HA_UPDATE_FROM_VERSION: ${{ needs.metadata.outputs.ha_update_from_version }}
HA_UPDATE_FROM_COMMIT: ${{ needs.metadata.outputs.ha_update_from_commit }}
VERSION: ${{ needs.metadata.outputs.version }}
steps:
- name: Checkout code
Expand Down Expand Up @@ -530,6 +559,10 @@ jobs:
echo "version: $VERSION" > deployment/version.txt
echo "build_date: $BUILD_DATE" >> deployment/version.txt
echo "commit: ${{ github.sha }}" >> deployment/version.txt
if [[ -n "$HA_UPDATE_FROM_VERSION" ]]; then
echo "ha_update_from: $HA_UPDATE_FROM_VERSION" >> deployment/version.txt
echo "ha_update_from_commit: $HA_UPDATE_FROM_COMMIT" >> deployment/version.txt
fi
if [[ "$CHANNEL" == "nightly" ]]; then
echo "channel: $CHANNEL" >> deployment/version.txt
fi
Expand Down
52 changes: 50 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,15 @@ permissions:
jobs:
validate-tag:
runs-on: ubuntu-latest
outputs:
ha_update_from_version: ${{ steps.ha-update-from.outputs.version }}
ha_update_from_commit: ${{ steps.ha-update-from.outputs.commit }}
steps:
- name: Checkout release history
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0

- name: Validate tag format
env:
TAG_NAME: ${{ github.ref_name }}
Expand Down Expand Up @@ -54,20 +62,60 @@ jobs:
fi
echo "Release/RC tag is on main — OK"

- name: Read qualified HA update source
id: ha-update-from
env:
GH_TOKEN: ${{ github.token }}
TAG_NAME: ${{ github.ref_name }}
run: |
qualified=$(sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' deployment-files/ha/qualified-update-from.txt)
# RCs remain clean-install-only. The first HA updater supports only
# qualified stable-to-stable transitions.
if [[ "$TAG_NAME" == *-* ]]; then
qualified=''
fi
if [[ $(printf '%s\n' "$qualified" | grep -c . || true) -gt 1 ]]; then
echo "::error::qualified-update-from.txt must contain at most one release tag"
exit 1
fi
if [[ -n "$qualified" ]]; then
if [[ ! "$qualified" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "::error::Qualified HA update source '$qualified' is not a stable release tag"
exit 1
fi
if [[ "$qualified" == "$TAG_NAME" ]] || ! git merge-base --is-ancestor "$qualified" "$GITHUB_SHA"; then
echo "::error::Qualified HA update source '$qualified' must be an existing ancestor tag"
exit 1
fi
previous=$(gh api --paginate "repos/${{ github.repository }}/releases?per_page=100" \
--jq '.[] | select(.draft == false and .prerelease == false and (.tag_name | test("^v[0-9]+\\.[0-9]+\\.[0-9]+$"))) | [.published_at, .tag_name] | @tsv' \
| sort -r | awk 'NR == 1 { print $2 }')
if [[ "$qualified" != "$previous" ]]; then
echo "::error::Qualified HA update source '$qualified' must be the latest published stable release '$previous'"
exit 1
fi
fi
echo "version=$qualified" >> "$GITHUB_OUTPUT"
if [[ -n "$qualified" ]]; then
echo "commit=$(git rev-list -n 1 "$qualified")" >> "$GITHUB_OUTPUT"
fi

build-artifacts:
name: Build release artifacts
needs: [validate-tag]
uses: ./.github/workflows/proto-fleet-artifact-build.yml
with:
version: ${{ github.ref_name }}
ha_update_from_version: ${{ needs.validate-tag.outputs.ha_update_from_version }}
ha_update_from_commit: ${{ needs.validate-tag.outputs.ha_update_from_commit }}
retention_days: 1
channel: release
is_prerelease: ${{ contains(github.ref_name, '-') }}
secrets: inherit

publish-proto-fleet:
runs-on: ubuntu-latest
needs: [build-artifacts]
needs: [validate-tag, build-artifacts]
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -132,7 +180,7 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG_NAME: ${{ github.ref_name }}
IS_PRERELEASE: ${{ contains(github.ref_name, '-') }}
IS_PRERELEASE: ${{ contains(github.ref_name, '-') || needs.validate-tag.outputs.ha_update_from_version != '' }}
Comment thread
ankitgoswami marked this conversation as resolved.
Comment thread
ankitgoswami marked this conversation as resolved.
run: |
set -euo pipefail
shopt -s failglob
Expand Down
28 changes: 28 additions & 0 deletions deployment-files/ha/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,3 +197,31 @@ Repeat on `ha-b` without `HA_PROFILE_MIGRATE`. The emitted
`HA_PROFILE_EVIDENCE` line proves that the deployment artifacts, etcd leader,
Patroni primary, and connected PostgreSQL writer agree.
The qualification workflow owns the complete failure matrix and evidence.

This first application-only updater supports one adjacent transition from a
clean-installed release. It rejects a chained update after the application has
advanced beyond the pinned database and DCS substrate. Every target migration
must be expand-only and remain compatible with the running release; destructive
contract migrations require a later release after both hosts have advanced. Do
not use `fleet-ha update` for a release pair that has not passed separate
adjacent-release migration and mixed-version qualification.

The first release containing this update workflow is the clean-install
baseline. HA deployments on an earlier experimental release must be reinstalled
instead of upgraded through this path. Release metadata allows no HA source by
default. To qualify an adjacent pair, set its one exact source tag in
`qualified-update-from.txt` before publishing the target. The manifest covers
the resulting source tag and commit, and the updater requires both to match the
installed release.

Production HA updates accept only a promoted GitHub release. Qualification may
exercise one exact prerelease by setting
`PROTO_FLEET_HA_QUALIFICATION_TARGET=vX.Y.Z` in the root-owned
`/etc/proto-fleet/updater.env` and restarting the updater. Remove the setting
after the qualification run. This is not a general prerelease update channel.
Only stable-to-stable pairs are supported; RC releases are clean-install-only.

A passive update rechecks the local role, active peer, and control path just
before stopping Fleet. This crash-only profile accepts the small role-change
window before process exit; if the peer fails in that interval, durable update
recovery restarts the local application.
6 changes: 6 additions & 0 deletions deployment-files/ha/ha-updater-systemd.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[Unit]
Wants=proto-fleet-updater.service

[Service]
EnvironmentFile=-/etc/proto-fleet/updater.env
ExecStartPre=/usr/local/libexec/proto-fleet/proto-fleet-updater --deployment-mode ha --self-update-path /usr/local/libexec/proto-fleet/proto-fleet-updater --repair-startup
Comment thread
ankitgoswami marked this conversation as resolved.
1 change: 1 addition & 0 deletions deployment-files/ha/qualified-update-from.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Leave empty for a clean-install-only release. Set one exact source tag before publishing a target for qualification.
9 changes: 9 additions & 0 deletions deployment-files/ha/tests/test-profile.sh
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ test_fleet_ha_contract() {
assert_contains "$rendered" "https://10.40.0.11:2379,https://10.40.0.12:2379,https://10.40.0.13:2379"
assert_contains "$rendered" "FLEET_HA_ENDPOINT_IP: 10.40.0.100"
assert_contains "$rendered" "FLEET_HA_ENDPOINT_INTERFACE: eth0"
assert_contains "$rendered" "UPDATES_ENABLED: \"false\""
assert_contains "$rendered" "sleep 15; exec /app/fleetd"
[[ "$(grep -c 'restart: on-failure' "$rendered")" -eq 2 ]] ||
fail "Fleet services must restart process failures without bypassing the systemd start gate"
Expand All @@ -134,6 +135,7 @@ test_fleet_ha_contract() {
secret_mount_count="$(grep -c 'source: /etc/proto-fleet/ha/' "$rendered")"
[[ "$secret_mount_count" -eq 4 ]] || fail "Fleet services must mount only their required HA secret files"
assert_not_contains "$rendered" "source: ${release_dir}/ssl"
assert_not_contains "$rendered" "/run/proto-fleet-updater"

assert_contains "${HA_DIR}/scripts/check-fleet-active.sh" '--cacert "$service_ca"'
assert_contains "${HA_DIR}/scripts/check-fleet-active.sh" '--connect-to "${virtual_ip}:443:127.0.0.1:443"'
Expand Down Expand Up @@ -171,6 +173,13 @@ test_fleet_ha_contract() {
assert_contains "${HA_DIR}/docker-systemd.conf" "PartOf=nftables.service"
assert_not_contains "${HA_DIR}/docker-systemd.conf" "Wants=proto-fleet-ha.service"
assert_contains "${HA_DIR}/docker-ha-recovery-systemd.conf" "Wants=proto-fleet-ha.service"
assert_contains "${HA_DIR}/updater-systemd.conf" "ReadWritePaths=/etc/proto-fleet/ha"
assert_contains "${HA_DIR}/updater-systemd.conf" "After=proto-fleet-ha.service"
assert_contains "${HA_DIR}/updater-systemd.conf" "PartOf=proto-fleet-ha.service"
assert_contains "${HA_DIR}/ha-updater-systemd.conf" "EnvironmentFile=-/etc/proto-fleet/updater.env"
assert_contains "${HA_DIR}/ha-updater-systemd.conf" "Wants=proto-fleet-updater.service"
assert_contains "${HA_DIR}/ha-updater-systemd.conf" "ExecStartPre=/usr/local/libexec/proto-fleet/proto-fleet-updater --deployment-mode ha --self-update-path /usr/local/libexec/proto-fleet/proto-fleet-updater --repair-startup"
assert_not_contains "${HA_DIR}/ha-updater-systemd.conf" "Requires=proto-fleet-updater.service"

for nginx_config in "${HA_DIR}/../client/nginx.http.conf" "${HA_DIR}/../client/nginx.https.conf"; do
assert_contains "$nginx_config" "location ^~ /api-proxy/health/ha"
Expand Down
6 changes: 6 additions & 0 deletions deployment-files/ha/updater-systemd.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[Unit]
After=proto-fleet-ha.service
PartOf=proto-fleet-ha.service

[Service]
ReadWritePaths=/etc/proto-fleet/ha
Loading
Loading