ci: build release binaries, and prove each one leaves its build machine - #21
Merged
Conversation
Linux uses cargo, not nix. The documented cargo route already produces an ordinary portable binary: linux-vdso, libgcc_s, libc.so.6 and the standard loader. Nix would give one whose interpreter and libraries live in /nix/store. It runs in a Debian 12 container because the glibc floor is set by the build host, not by this code, and ubuntu-latest would stamp GLIBC_2.39 on the asset and lock out Ubuntu 22.04, Debian 12 and RHEL 9. Debian 12 gives GLIBC_2.34. macOS uses nix, because cargo cannot build there at all. But a nix-built macOS binary is not relocatable as it comes out: measured on a real artifact, otool -L reported an absolute /nix/store path for libiconv. Shipping that gives a file that runs for us and fails on every machine without that exact store. So the load commands are rewritten to the system copies, any store rpath is deleted, the binary is re-signed because install_name_tool invalidates the signature, and then it is GATED: the job fails if any /nix/store reference survives. A release asset that only runs where it was built is worse than no asset, because it fails at the user instead of at the build. Both jobs prove the binary runs before they upload it. A dispatch run stops after the two build jobs, so an asset can be proved before anything is published. Only a v* tag publishes. The branch trigger is temporary and comes out before merge. It is there because a workflow_dispatch-only workflow is not dispatchable until it is on the default branch, and merging a release workflow nobody has run is the wrong order.
The first Linux asset reported 0.13.0-rust+unknown. The macOS one reported the real sha, which is what made the cause obvious: build.rs derives the sha by running git in the checkout, and git refused inside the Debian container because actions/checkout and the container disagree about directory ownership. On the macOS runner there is no container and git worked. Everything else about that asset was right: GLIBC_2.34 as intended, only libgcc_s and libc, checksum matching, and it ran. It simply could not say what it was. That is not cosmetic. It is the exact failure this rollout exists to fix. A binary had been serving a machine for five days and the only way anyone found out which build it was, was by reading its version string. An asset that answers "unknown" cannot be used to decide whether a machine is up to date. PTY_BUILD_SHA is build.rs's own escape hatch and takes precedence over git, so setting it from github.sha makes the version independent of whether git works in any given environment. Both jobs set it, not just the one that failed: the macOS job worked by luck of not being in a container, and that is not a reason to leave it depending on luck. Both jobs now also REFUSE to stage an asset whose version contains "unknown". The fix would otherwise be invisible the next time something breaks it.
The Linux job failed with "Bad substitution". The container runs `sh -e`, and
${GITHUB_SHA::7} is a bashism; the macOS runner uses bash and accepted it.
That is the same asymmetry as the bug the step exists to fix, one layer up. I
wrote a fix for a platform-dependent failure and the fix was itself platform
dependent, passing on the runner that had never had the original problem.
`echo "$GITHUB_SHA" | cut -c1-7` works in both. Verified in dash and bash side
by side: the cut form gives ac91460 in each, and the substring form gives
ac91460 in bash and "Bad substitution" in sh.
Both jobs changed, again, rather than the one that failed.
It was there so the workflow could be run before it was merged, because a workflow_dispatch-only workflow is not dispatchable until it exists on the default branch, and merging a release workflow nobody has run is the wrong order. It has now run four times and produced two assets I downloaded and checked myself, so it comes out as promised. Releases are cut by pushing a v* tag. A workflow_dispatch run builds and verifies both assets without publishing.
Twice in one afternoon a fix for a platform-dependent failure was itself platform-dependent, and passed on the platform that never had the problem. The unknown version, then the bashism in the fix for it. The rule that falls out is not about cut versus substring expansion. It is that a green job on one platform says nothing about the other, so a fix goes into both jobs and is verified in both shells even when only one was broken. Recorded above the job list, where someone writing a two-platform workflow will read it, with both examples, because the examples are what make it believable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nathan asked for binaries built in Actions and attached to releases. This is that, plus the checks that make an asset worth attaching.
Two platforms, two different routes, for measured reasons
Linux uses cargo, not nix. The documented cargo route already produces an ordinary portable binary —
linux-vdso,libgcc_s,libc.so.6, the standard loader. Nix would produce one whose interpreter and libraries live in/nix/store.It builds in a Debian 12 container because the glibc floor is set by the build host, not by this code:
ubuntu-latestwould stampGLIBC_2.39on the asset and lock out Ubuntu 22.04, Debian 12 and RHEL 9. Debian 12 givesGLIBC_2.34, which covers them. Zig is pinned by version and sha256.macOS uses nix, because cargo cannot build there at all — ghostty pins Zig 0.15.2 and that Zig cannot link the macOS 26.5 SDK. But a nix-built macOS binary is not relocatable as it comes out, so the job undoes that: store dylib references are rewritten to their
/usr/libcounterparts, any storeLC_RPATHis deleted, and the binary is re-signed becauseinstall_name_toolinvalidates the signature and Apple silicon will not run a mis-signed one.Measured in the job, on the same store hash seen on a real machine:
Every asset is gated, not trusted
/nix/store;--versionreportsunknown;Two defects this caught in its own output, which is why the gates are there
0.13.0-rust+unknown. The first Linux asset could not say which commit it was.build.rsderives the sha by running git, and git refused inside the container over directory ownership; the macOS job got the real sha because it has no container. That is not cosmetic — it is the exact failure this work exists to fix, since a binary served a machine for five days and its version string was the only way to find out which build it was. Fixed withPTY_BUILD_SHA, and now gated.Bad substitution. The fix for that was itself platform-dependent:${GITHUB_SHA::7}is a bashism, the container runssh, and the macOS runner accepted it. Same asymmetry one layer up. Nowcut, verified in dash and bash side by side.Both times the fix went into both jobs rather than the one that failed, because a job passing by luck of its environment is how the next failure looks like a new bug.
Verified by download, not by job summary
Both assets pulled and checked locally: checksums match, both report
0.13.0-rust+01af50e, the Linux one runs atGLIBC_2.34with onlylibgcc_sandlibc, and the macOS one contains zero/nix/storestrings.How a release is cut
Push a
v*tag. Aworkflow_dispatchrun builds and verifies both assets without publishing, so an asset can be proved before anything is public.The temporary branch trigger that let this run before merging is removed.