Skip to content

ci: cross-compile Go builds instead of QEMU emulation#382

Merged
huang195 merged 1 commit into
rossoctl:mainfrom
huang195:ci/cross-compile-go-builds
May 27, 2026
Merged

ci: cross-compile Go builds instead of QEMU emulation#382
huang195 merged 1 commit into
rossoctl:mainfrom
huang195:ci/cross-compile-go-builds

Conversation

@huang195

Copy link
Copy Markdown
Member

Summary

  • Add --platform=$BUILDPLATFORM to the builder FROM in all three Dockerfiles (operator, agentcard-signer, test-tls-agent) and drop -a from go build. Pins the builder stage to the host arch so Go cross-compiles to $TARGETARCH instead of running under QEMU.
  • Investigation of the latest release run showed the linux/arm64 builder step at 1335s (22m15s) vs linux/amd64 at 155s native — a ~9× slowdown caused entirely by emulating the arm64 CPU through tonistiigi/binfmt. CGO is disabled and there are no per-arch dependencies, so cross-compiling produces a bit-for-bit identical binary.
  • Existing GOARCH=${TARGETARCH} plumbing already wires up cross-compile correctly; only the runtime of the builder stage was wrong.
  • -a forces rebuild of every package including stdlib, defeating the build cache. The Makefile's build target already omits it; the Dockerfiles were the outliers. Modern kubebuilder v4 scaffolds (this project's PROJECT layout) ship with both fixes by default.

Expected impact

Pipeline Before After (estimate)
release workflow (per-image, parallel matrix) ~25 min ~5 min cold cache
linux/arm64 builder step (Go compile) ~22 min ~2–3 min

Test plan

  • CI release workflow runs to completion on this PR / merge.
  • docker buildx imagetools inspect ghcr.io/kagenti/kagenti-operator/kagenti-operator:latest after merge reports both linux/amd64 and linux/arm64 digests (parity with current behavior).
  • e2e-helm-install job (Kind on amd64) still passes — pulls the same multi-arch manifest, only the build path for amd64 changed (and amd64 was already native, so it should be a no-op for that arch).
  • Spot-check binary determinism: pull and sha256sum the manager binary out of the new image vs. a prior release; bytes should match for the same toolchain version (Go's content-addressable cache makes -a irrelevant for output bytes).

Notes

  • No workflow file changes needed — this is purely a Dockerfile change.
  • Same change applies cleanly to the kagenti-webhook Dockerfile in akram-kagenti-extensions if/when that repo is upstreamed.

Assisted-By: Claude Code

Add --platform=$BUILDPLATFORM to the builder FROM in all three
Dockerfiles and drop the -a flag from go build. This pins the
builder stage to the host arch so Go cross-compiles to $TARGETARCH
instead of running under QEMU.

Effect on the release pipeline (kagenti-operator image, latest run):
the linux/arm64 builder step takes 22m15s under emulation vs ~155s
amd64 native — a ~9x slowdown. Cross-compiling is bit-for-bit
identical (CGO_ENABLED=0, no per-arch dependencies) and the existing
GOARCH=${TARGETARCH} plumbing already wires up cross-compile
correctly; only the runtime of the builder stage was wrong.

The -a flag forces rebuild of every package including stdlib,
defeating the build cache. The Makefile already drops it for local
builds; the Dockerfiles were the outliers. Modern kubebuilder v4
scaffolds (this project's PROJECT layout) ship with both fixes by
default — this brings the Dockerfiles in line with the upstream
template.

Expected total CI time for the release workflow: ~25min -> ~5min on
cold cache.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>

@cwiklik cwiklik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, consistent change across all three Dockerfiles. --platform=$BUILDPLATFORM pins the builder to host arch for native cross-compilation instead of QEMU emulation, and dropping -a restores build cache. The existing GOARCH=${TARGETARCH} plumbing was already correct — only the builder runtime was wrong.

One nit inline about a stale comment (non-blocking).

@@ -21,7 +26,7 @@ COPY internal/ internal/
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This comment ("by leaving it empty we can ensure that the container and binary shipped on it will have the same platform") is now stale — it described the old approach of not pinning the builder platform. With --platform=$BUILDPLATFORM on line 7, the builder IS explicitly pinned to host arch and Go cross-compiles via GOARCH=${TARGETARCH}. The new comment at the top of the file explains the correct rationale. Consider removing or updating these three lines to avoid contradicting the new approach.

@huang195
huang195 merged commit 2073d03 into rossoctl:main May 27, 2026
16 checks passed
@github-project-automation github-project-automation Bot moved this from New /:ToDo to Done in Rossoctl Issue Prioritization May 27, 2026
@huang195
huang195 deleted the ci/cross-compile-go-builds branch May 27, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants