ci: reuse cached signed/notarized native binaries on release (#775)

theoephraim · web-flow · commit 8a4a91dd2f96 · 2026-06-11T22:14:11.000-07:00
* ci: reuse cached signed/notarized native binaries on release

The release workflow rebuilt and re-notarized the macOS/Rust native
encryption binaries on every publish, even when nothing under the native
source changed. The Apple notarization round-trip in particular is slow
and externally dependent, and re-running it for a byte-identical binary
is wasted work.

Gate the native build/notarize/Rust jobs on a content-addressed cache
probe. The cache key covers the native source, the build/bundle scripts,
and the workflows that drive them, plus a manual 'v1' salt for
toolchain bumps the file hash can't see. On a cache hit the build and
notarize jobs are skipped and the previously notarized bundle is reused.

To never publish the wrong thing:
- keys are content-addressed, so a restored binary always matches the
  source that produced it
- caches restored on a release are main-scoped only (feature-branch
  caches are not visible), so a release can only reuse a binary a prior
  main run produced
- a reused macOS bundle is re-verified on a macOS runner (codesign +
  notarization staple + functional smoke) before the publish step; if it
  can't be verified the release is blocked
- the publish job fails loudly if any native binary is missing

First release after this lands rebuilds once to seed the cache, then
subsequent releases with unchanged native source reuse it.

* ci: hardcode native bundle version constant to enforce byte-stability

The macOS .app reuse relied on the bundle being byte-identical across
releases, but that was only true by accident: release.yaml happened not
to pass a --version, so the bundle fell back to a constant default.
Anyone wiring a real version into that call would have silently defeated
the notarized-bundle cache with no error.

Make it stable by construction. CFBundleVersion is vestigial — nothing
reads it (Swift binary, Rust binary, varlock CLI, notarization) — so
hardcode it to a fixed constant in build-swift.ts and remove the
--version plumbing from the macOS build workflow and its callers. The
bundle is now byte-stable in every path, not just the release one.

First release after this rebuilds + re-notarizes once (the build script
hash changed), then reuse kicks in.

* ci: drop orphaned version plumbing from binary-release debug job

The bundle version is now a fixed constant (build-swift.ts), so the
binary-release path no longer threads a version into the macOS build.
That left the debug job's version output unused and a vestigial
'needs: debug' on build-native-macos. Remove both; the context-printing
debug step stays. The re-release flow (workflow_dispatch version input -&gt;
RELEASE_VERSION -&gt; release upload / homebrew / docker) is unchanged.
diff --git a/.github/workflows/binary-release.yaml b/.github/workflows/binary-release.yaml
@@ -20,32 +20,19 @@ concurrency: ${{ github.workflow }}-${{ github.ref }}
 jobs:
   debug:
     runs-on: ubuntu-latest
-    outputs:
-      version: ${{ steps.version.outputs.version }}
     steps:
       - name: print github context
         env:
           GITHUB_CONTEXT: ${{ toJson(github) }}
         run: |
           echo "$GITHUB_CONTEXT"
-      - name: Extract version
-        id: version
-        run: |
-          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            echo "version=${{ inputs.version }}" >> $GITHUB_OUTPUT
-          else
-            # github.ref_name is e.g. "varlock@1.2.3" — strip the package prefix
-            echo "version=${GITHUB_REF_NAME#varlock@}" >> $GITHUB_OUTPUT
-          fi
 
   # Build and sign the macOS native binary (cache hit if already built in CI)
   build-native-macos:
-    needs: debug
     if: github.event_name == 'workflow_dispatch' || startsWith(github.ref_name, 'varlock@')
     uses: ./.github/workflows/build-native-macos.yaml
     with:
       mode: release
-      version: ${{ needs.debug.outputs.version }}
       artifact-name: native-bin-macos-signed
     secrets:
       OP_CI_TOKEN: ${{ secrets.OP_CI_TOKEN }}
diff --git a/.github/workflows/build-native-macos.yaml b/.github/workflows/build-native-macos.yaml
@@ -5,7 +5,7 @@ name: Build macOS native binary
 #
 # The Swift .build directory is cached by source hash, so the compile
 # step (~minutes) is near-instant on cache hit. The .app bundle wrapping
-# (plist, icon, signing) always runs since it varies by mode/version.
+# (plist, icon, signing) always runs since it varies by mode.
 #
 # Notarization is intentionally NOT included here — it's a separate
 # workflow for production releases.
@@ -20,10 +20,6 @@ on:
         description: 'Build mode: dev, preview, or release (affects bundle metadata)'
         type: string
         default: 'preview'
-      version:
-        description: 'Bundle version string (e.g. 1.2.3)'
-        type: string
-        default: '0.0.0-preview'
       artifact-name:
         description: 'Name for the uploaded artifact'
         type: string
@@ -103,7 +99,7 @@ jobs:
       - name: Build, bundle, and sign
         run: |
           bun run --filter @varlock/encryption-binary-swift build:universal \
-            -- --mode ${{ inputs.mode }} --version ${{ inputs.version }} --sign "$APPLE_SIGNING_IDENTITY"
+            -- --mode ${{ inputs.mode }} --sign "$APPLE_SIGNING_IDENTITY"
 
       - name: Verify binary
         run: |
diff --git a/.github/workflows/notarize-native-macos.yaml b/.github/workflows/notarize-native-macos.yaml
@@ -18,6 +18,10 @@ on:
         description: 'Name for the notarized artifact'
         type: string
         default: 'native-bin-macos-notarized'
+      cache-key:
+        description: 'If set, the notarized .app is saved to the Actions cache under this key for cross-run reuse'
+        type: string
+        default: ''
     secrets:
       OP_CI_TOKEN:
         required: true
@@ -89,3 +93,23 @@ jobs:
           name: ${{ inputs.artifact-name }}
           path: VarlockEnclave.app
           retention-days: 7
+
+      # Stage the notarized bundle at the canonical path and cache it so future
+      # releases with unchanged source can reuse it (skipping build + notarize).
+      # ditto preserves the bundle's signature and stapled notarization ticket.
+      # The cache key is content-addressed (source + scripts + workflows), so a
+      # restored bundle always corresponds to the source that produced it; the
+      # release re-verifies it on macOS before publishing.
+      - name: Stage notarized .app for caching
+        if: inputs.cache-key != ''
+        run: |
+          mkdir -p packages/varlock/native-bins/darwin
+          rm -rf packages/varlock/native-bins/darwin/VarlockEnclave.app
+          ditto VarlockEnclave.app packages/varlock/native-bins/darwin/VarlockEnclave.app
+      - name: Cache notarized .app for cross-run reuse
+        if: inputs.cache-key != ''
+        uses: actions/cache/save@v5
+        continue-on-error: true  # immutable key; harmless if a concurrent run saved it first
+        with:
+          path: packages/varlock/native-bins/darwin/VarlockEnclave.app
+          key: ${{ inputs.cache-key }}
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -38,13 +38,20 @@ jobs:
           echo "::error::Timed out waiting for CI test suite"
           exit 1
 
-  # Determine what bumpy will do — only run expensive build/notarize when publishing
+  # Determine what bumpy will do — only run expensive build/notarize when publishing.
+  # Also probe the native-binary caches: if this exact source (+ build scripts +
+  # workflows) already produced a signed/notarized binary on a previous run, we
+  # reuse it instead of rebuilding and — for macOS — re-notarizing.
   plan:
     needs: wait-for-tests
     runs-on: ubuntu-latest
     outputs:
       mode: ${{ steps.plan.outputs.mode }}
       includes-varlock: ${{ contains(fromJSON(steps.plan.outputs.json).packageNames, 'varlock') }}
+      macos-cache-key: ${{ steps.keys.outputs.macos-cache-key }}
+      rust-source-hash: ${{ steps.keys.outputs.rust-source-hash }}
+      macos-cache-hit: ${{ steps.macos-cache.outputs.cache-hit }}
+      rust-cache-hit: ${{ steps.rust-cache.outputs.cache-hit }}
     steps:
       - uses: actions/checkout@v6
       - name: Setup Bun
@@ -54,39 +61,139 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
 
-  # Build and sign the macOS native binary (cache hit if already built in CI)
+      # Comprehensive content hashes for the native binaries. These cover the
+      # native source, the build/bundle scripts, AND the workflows that drive
+      # them — anything that can change the produced binary. The leading `v1`
+      # salt forces a full rebuild when something hashFiles can't see changes
+      # (runner image / Xcode / Rust / UPX toolchain) — bump it to invalidate.
+      #
+      # Note: the macOS .app bundle embeds a fixed constant version (hardcoded in
+      # build-swift.ts, not the npm version), so the notarized bundle is
+      # byte-stable across releases — which is what makes cross-release reuse
+      # valid. Do not reintroduce a per-release version there.
+      - id: keys
+        run: |
+          MACOS_HASH=${{ hashFiles('packages/encryption-binary-swift/swift/**', 'packages/encryption-binary-swift/scripts/**', 'packages/encryption-binary-swift/package.json', '.github/workflows/build-native-macos.yaml', '.github/workflows/notarize-native-macos.yaml') }}
+          RUST_HASH=${{ hashFiles('packages/encryption-binary-rust/Cargo.lock', 'packages/encryption-binary-rust/Cargo.toml', 'packages/encryption-binary-rust/src/**', 'packages/encryption-binary-rust/scripts/**', '.github/workflows/build-native-rust.yaml') }}
+          echo "macos-cache-key=native-bin-macos-notarized-v1-$MACOS_HASH" >> "$GITHUB_OUTPUT"
+          echo "rust-source-hash=v1-$RUST_HASH" >> "$GITHUB_OUTPUT"
+
+      # Probe caches (lookup-only, no download). cache-hit is 'true' only on an
+      # exact key match within this branch's scope (main + its base) — feature
+      # branch caches are not visible here, so a release can only reuse a binary
+      # produced by a prior run on main.
+      - name: Probe macOS notarized binary cache
+        id: macos-cache
+        if: steps.plan.outputs.mode == 'publish'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/darwin/VarlockEnclave.app
+          key: ${{ steps.keys.outputs.macos-cache-key }}
+          lookup-only: true
+      - name: Probe Rust cache - linux-x64
+        id: rust-cache-linux-x64
+        if: steps.plan.outputs.mode == 'publish'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/linux-x64/
+          key: native-bin-rust-linux-x64-${{ steps.keys.outputs.rust-source-hash }}
+          lookup-only: true
+      - name: Probe Rust cache - linux-arm64
+        id: rust-cache-linux-arm64
+        if: steps.plan.outputs.mode == 'publish'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/linux-arm64/
+          key: native-bin-rust-linux-arm64-${{ steps.keys.outputs.rust-source-hash }}
+          lookup-only: true
+      - name: Probe Rust cache - win32-x64
+        id: rust-cache-win32-x64
+        if: steps.plan.outputs.mode == 'publish'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/win32-x64/
+          key: native-bin-rust-win32-x64-${{ steps.keys.outputs.rust-source-hash }}
+          lookup-only: true
+      - name: Aggregate Rust cache status
+        id: rust-cache
+        if: steps.plan.outputs.mode == 'publish'
+        run: |
+          if [[ "${{ steps.rust-cache-linux-x64.outputs.cache-hit }}" == "true" \
+             && "${{ steps.rust-cache-linux-arm64.outputs.cache-hit }}" == "true" \
+             && "${{ steps.rust-cache-win32-x64.outputs.cache-hit }}" == "true" ]]; then
+            echo "cache-hit=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "cache-hit=false" >> "$GITHUB_OUTPUT"
+          fi
+
+  # Build + sign the macOS native binary. Skipped on a cache hit — the previously
+  # notarized bundle is reused and re-verified (see verify-native-macos) instead.
   build-native-macos:
     needs: plan
-    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.macos-cache-hit != 'true'
     uses: ./.github/workflows/build-native-macos.yaml
     with:
       mode: release
       artifact-name: native-bin-macos-signed
     secrets:
       OP_CI_TOKEN: ${{ secrets.OP_CI_TOKEN }}
 
-  # Notarize for production npm distribution
+  # Notarize for production npm distribution and cache the notarized bundle so
+  # future releases with unchanged source can reuse it.
   notarize-native-macos:
     needs: [plan, build-native-macos]
-    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.macos-cache-hit != 'true'
     uses: ./.github/workflows/notarize-native-macos.yaml
     with:
       source-artifact-name: native-bin-macos-signed
       artifact-name: native-bin-macos-npm
+      cache-key: ${{ needs.plan.outputs.macos-cache-key }}
     secrets:
       OP_CI_TOKEN: ${{ secrets.OP_CI_TOKEN }}
 
-  # Build Rust native binaries for Linux and Windows
+  # When the notarized bundle is reused from cache (build + notarize skipped),
+  # re-verify it on a macOS runner before we trust it in the publish step. This
+  # is the safety net for "never publish the wrong thing": a corrupted or
+  # un-notarized cached bundle fails here and blocks the release.
+  verify-native-macos:
+    needs: plan
+    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.macos-cache-hit == 'true'
+    runs-on: macos-latest
+    steps:
+      - name: Restore notarized macOS binary from cache
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/darwin/VarlockEnclave.app
+          key: ${{ needs.plan.outputs.macos-cache-key }}
+          fail-on-cache-miss: true
+      - name: Verify signature, notarization, and function
+        run: |
+          APP="packages/varlock/native-bins/darwin/VarlockEnclave.app"
+          BIN="$APP/Contents/MacOS/varlock-local-encrypt"
+          echo "=== codesign verify ==="
+          codesign --verify --deep --strict --verbose=2 "$APP"
+          codesign -dvv "$APP" 2>&1 || true
+          echo "=== notarization staple ==="
+          xcrun stapler validate "$APP"
+          echo "=== gatekeeper assessment (non-fatal) ==="
+          spctl -a -t exec -vv "$APP" || true
+          echo "=== functional smoke ==="
+          chmod +x "$BIN"
+          "$BIN" status | python3 -c "import sys,json; d=json.load(sys.stdin); assert d['ok'], 'status not ok'"
+          echo "Cached notarized binary verified"
+
+  # Build Rust native binaries for Linux and Windows. Skipped on a cache hit.
   build-native-rust:
     needs: plan
-    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+    if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit != 'true'
     uses: ./.github/workflows/build-native-rust.yaml
     with:
       artifact-name: native-bin-rust
+      source-hash: ${{ needs.plan.outputs.rust-source-hash }}
 
   release:
     name: Release
-    needs: [plan, notarize-native-macos, build-native-rust]
+    needs: [plan, notarize-native-macos, verify-native-macos, build-native-rust]
     if: always() && !failure() && !cancelled()
     runs-on: ubuntu-latest
     permissions:
@@ -117,42 +224,86 @@ jobs:
       - name: Update npm
         run: npm install -g npm@latest
 
-      # Download signed macOS native binary so it's included in the npm package
-      - name: Download macOS native binary
-        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+      # macOS native binary: download the freshly-notarized artifact (cache miss),
+      # or restore the previously-notarized bundle from cache (cache hit, already
+      # re-verified by verify-native-macos).
+      - name: Download macOS native binary (built this run)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.macos-cache-hit != 'true'
         uses: actions/download-artifact@v8
         with:
           name: native-bin-macos-npm
           path: packages/varlock/native-bins/darwin/VarlockEnclave.app
+      - name: Restore macOS native binary (from cache)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.macos-cache-hit == 'true'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/darwin/VarlockEnclave.app
+          key: ${{ needs.plan.outputs.macos-cache-key }}
+          fail-on-cache-miss: true
       - name: Restore native binary execute permission
         if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
         run: chmod +x packages/varlock/native-bins/darwin/VarlockEnclave.app/Contents/MacOS/varlock-local-encrypt
 
-      # Download Rust native binaries for Linux and Windows
-      - name: Download Linux x64 native binary
-        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+      # Rust native binaries: download artifacts (cache miss) or restore from cache (hit)
+      - name: Download Rust native binaries (built this run)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit != 'true'
         uses: actions/download-artifact@v8
         with:
           name: native-bin-rust-linux-x64
           path: packages/varlock/native-bins/linux-x64
-      - name: Download Linux arm64 native binary
-        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+      - name: Download Rust native binaries - linux-arm64 (built this run)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit != 'true'
         uses: actions/download-artifact@v8
         with:
           name: native-bin-rust-linux-arm64
           path: packages/varlock/native-bins/linux-arm64
-      - name: Download Windows x64 native binary
-        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+      - name: Download Rust native binaries - win32-x64 (built this run)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit != 'true'
         uses: actions/download-artifact@v8
         with:
           name: native-bin-rust-win32-x64
           path: packages/varlock/native-bins/win32-x64
+      - name: Restore Rust native binaries (from cache)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit == 'true'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/linux-x64/
+          key: native-bin-rust-linux-x64-${{ needs.plan.outputs.rust-source-hash }}
+          fail-on-cache-miss: true
+      - name: Restore Rust native binaries - linux-arm64 (from cache)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit == 'true'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/linux-arm64/
+          key: native-bin-rust-linux-arm64-${{ needs.plan.outputs.rust-source-hash }}
+          fail-on-cache-miss: true
+      - name: Restore Rust native binaries - win32-x64 (from cache)
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true' && needs.plan.outputs.rust-cache-hit == 'true'
+        uses: actions/cache/restore@v5
+        with:
+          path: packages/varlock/native-bins/win32-x64/
+          key: native-bin-rust-win32-x64-${{ needs.plan.outputs.rust-source-hash }}
+          fail-on-cache-miss: true
       - name: Restore Rust binary execute permissions
         if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
         run: |
           chmod +x packages/varlock/native-bins/linux-x64/varlock-local-encrypt
           chmod +x packages/varlock/native-bins/linux-arm64/varlock-local-encrypt
 
+      # Fail loudly if any native binary is missing before we build the npm package,
+      # rather than silently publishing an incomplete package.
+      - name: Verify native binaries present
+        if: needs.plan.outputs.mode == 'publish' && needs.plan.outputs.includes-varlock == 'true'
+        run: |
+          missing=0
+          check() { if [ ! -f "$1" ]; then echo "::error::missing native binary: $1"; missing=1; fi; }
+          check packages/varlock/native-bins/darwin/VarlockEnclave.app/Contents/MacOS/varlock-local-encrypt
+          check packages/varlock/native-bins/linux-x64/varlock-local-encrypt
+          check packages/varlock/native-bins/linux-arm64/varlock-local-encrypt
+          check packages/varlock/native-bins/win32-x64/varlock-local-encrypt.exe
+          if [ "$missing" = "1" ]; then exit 1; fi
+          echo "All native binaries present"
+
       # ------------------------------------------------------------
       - name: Build libraries
         run: bun run build:libs
diff --git a/packages/encryption-binary-swift/scripts/build-swift.ts b/packages/encryption-binary-swift/scripts/build-swift.ts
