Skip to content

ci: build release binaries, and prove each one leaves its build machine - #21

Merged
myobie merged 5 commits into
mainfrom
feat/release-binaries
Sep 7, 2026
Merged

ci: build release binaries, and prove each one leaves its build machine#21
myobie merged 5 commits into
mainfrom
feat/release-binaries

Conversation

@myobie

@myobie myobie commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Nathan asked for binaries built in Actions and attached to releases. This is that, plus the checks that make an asset worth attaching.

Two platforms, two different routes, for measured reasons

Linux uses cargo, not nix. The documented cargo route already produces an ordinary portable binary — linux-vdso, libgcc_s, libc.so.6, the standard loader. Nix would produce one whose interpreter and libraries live in /nix/store.

It builds in a Debian 12 container because the glibc floor is set by the build host, not by this code: ubuntu-latest would stamp GLIBC_2.39 on the asset and lock out Ubuntu 22.04, Debian 12 and RHEL 9. Debian 12 gives GLIBC_2.34, which covers them. Zig is pinned by version and sha256.

macOS uses nix, because cargo cannot build there at all — ghostty pins Zig 0.15.2 and that Zig cannot link the macOS 26.5 SDK. But a nix-built macOS binary is not relocatable as it comes out, so the job undoes that: store dylib references are rewritten to their /usr/lib counterparts, any store LC_RPATH is deleted, and the binary is re-signed because install_name_tool invalidates the signature and Apple silicon will not run a mis-signed one.

Measured in the job, on the same store hash seen on a real machine:

before:  /nix/store/0ky902fx…-libiconv-115.100.1/lib/libiconv.2.dylib
after:   /usr/lib/libiconv.2.dylib
         no /nix/store references remain
         dist/pty: satisfies its Designated Requirement

Every asset is gated, not trusted

  • fails if the binary references /nix/store;
  • fails if --version reports unknown;
  • runs the binary before staging it;
  • publishes a sha256 beside each asset.

Two defects this caught in its own output, which is why the gates are there

0.13.0-rust+unknown. The first Linux asset could not say which commit it was. build.rs derives the sha by running git, and git refused inside the container over directory ownership; the macOS job got the real sha because it has no container. That is not cosmetic — it is the exact failure this work exists to fix, since a binary served a machine for five days and its version string was the only way to find out which build it was. Fixed with PTY_BUILD_SHA, and now gated.

Bad substitution. The fix for that was itself platform-dependent: ${GITHUB_SHA::7} is a bashism, the container runs sh, and the macOS runner accepted it. Same asymmetry one layer up. Now cut, verified in dash and bash side by side.

Both times the fix went into both jobs rather than the one that failed, because a job passing by luck of its environment is how the next failure looks like a new bug.

Verified by download, not by job summary

Both assets pulled and checked locally: checksums match, both report 0.13.0-rust+01af50e, the Linux one runs at GLIBC_2.34 with only libgcc_s and libc, and the macOS one contains zero /nix/store strings.

How a release is cut

Push a v* tag. A workflow_dispatch run builds and verifies both assets without publishing, so an asset can be proved before anything is public.

The temporary branch trigger that let this run before merging is removed.

Linux uses cargo, not nix. The documented cargo route already produces an
ordinary portable binary: linux-vdso, libgcc_s, libc.so.6 and the standard
loader. Nix would give one whose interpreter and libraries live in /nix/store.
It runs in a Debian 12 container because the glibc floor is set by the build
host, not by this code, and ubuntu-latest would stamp GLIBC_2.39 on the asset
and lock out Ubuntu 22.04, Debian 12 and RHEL 9. Debian 12 gives GLIBC_2.34.

macOS uses nix, because cargo cannot build there at all. But a nix-built macOS
binary is not relocatable as it comes out: measured on a real artifact, otool -L
reported an absolute /nix/store path for libiconv. Shipping that gives a file
that runs for us and fails on every machine without that exact store. So the
load commands are rewritten to the system copies, any store rpath is deleted,
the binary is re-signed because install_name_tool invalidates the signature, and
then it is GATED: the job fails if any /nix/store reference survives.

A release asset that only runs where it was built is worse than no asset,
because it fails at the user instead of at the build. Both jobs prove the
binary runs before they upload it.

A dispatch run stops after the two build jobs, so an asset can be proved before
anything is published. Only a v* tag publishes.

The branch trigger is temporary and comes out before merge. It is there because
a workflow_dispatch-only workflow is not dispatchable until it is on the default
branch, and merging a release workflow nobody has run is the wrong order.
The first Linux asset reported 0.13.0-rust+unknown. The macOS one reported the
real sha, which is what made the cause obvious: build.rs derives the sha by
running git in the checkout, and git refused inside the Debian container
because actions/checkout and the container disagree about directory ownership.
On the macOS runner there is no container and git worked.

Everything else about that asset was right: GLIBC_2.34 as intended, only
libgcc_s and libc, checksum matching, and it ran. It simply could not say what
it was.

That is not cosmetic. It is the exact failure this rollout exists to fix. A
binary had been serving a machine for five days and the only way anyone found
out which build it was, was by reading its version string. An asset that
answers "unknown" cannot be used to decide whether a machine is up to date.

PTY_BUILD_SHA is build.rs's own escape hatch and takes precedence over git, so
setting it from github.sha makes the version independent of whether git works in
any given environment. Both jobs set it, not just the one that failed: the macOS
job worked by luck of not being in a container, and that is not a reason to
leave it depending on luck.

Both jobs now also REFUSE to stage an asset whose version contains "unknown".
The fix would otherwise be invisible the next time something breaks it.
The Linux job failed with "Bad substitution". The container runs `sh -e`, and
${GITHUB_SHA::7} is a bashism; the macOS runner uses bash and accepted it.

That is the same asymmetry as the bug the step exists to fix, one layer up. I
wrote a fix for a platform-dependent failure and the fix was itself platform
dependent, passing on the runner that had never had the original problem.

`echo "$GITHUB_SHA" | cut -c1-7` works in both. Verified in dash and bash side
by side: the cut form gives ac91460 in each, and the substring form gives
ac91460 in bash and "Bad substitution" in sh.

Both jobs changed, again, rather than the one that failed.
It was there so the workflow could be run before it was merged, because a
workflow_dispatch-only workflow is not dispatchable until it exists on the
default branch, and merging a release workflow nobody has run is the wrong
order. It has now run four times and produced two assets I downloaded and
checked myself, so it comes out as promised.

Releases are cut by pushing a v* tag. A workflow_dispatch run builds and
verifies both assets without publishing.
Twice in one afternoon a fix for a platform-dependent failure was itself
platform-dependent, and passed on the platform that never had the problem. The
unknown version, then the bashism in the fix for it.

The rule that falls out is not about cut versus substring expansion. It is that
a green job on one platform says nothing about the other, so a fix goes into
both jobs and is verified in both shells even when only one was broken.

Recorded above the job list, where someone writing a two-platform workflow will
read it, with both examples, because the examples are what make it believable.
@myobie
myobie merged commit 15ca74f into main Sep 7, 2026
7 checks passed
@myobie
myobie deleted the feat/release-binaries branch September 7, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant