fix: add image-manifest=true to BuildKit cache export for ephemeral VMs#83
Merged
hiroTamada merged 4 commits intomainfrom Feb 6, 2026
Merged
fix: add image-manifest=true to BuildKit cache export for ephemeral VMs#83hiroTamada merged 4 commits intomainfrom
hiroTamada merged 4 commits intomainfrom
Conversation
Without image-manifest=true, BuildKit's registry cache stores layer references pointing to external registries (e.g., docker.io) rather than copying the actual layer blobs into the cache image. This causes cache misses in ephemeral BuildKit instances (like our builder VMs) because the layers aren't available locally. With image-manifest=true, BuildKit creates a proper OCI image manifest with all layer blobs stored in the registry, enabling cache hits even in fresh BuildKit instances. This fixes the issue where the global cache (populated by admin builds) wasn't providing cache hits for tenant builds - the first deployment for each tenant was re-downloading all base image layers from Docker Hub. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a unit test that reproduces the production issue where hypeman fails to pre-pull BuildKit cache images. The test creates a mock OCI layout with BuildKit's cache config mediatype (application/vnd.buildkit.cacheconfig.v0) and verifies that unpackLayers fails with the expected error. This test documents the root cause: umoci expects standard OCI config mediatype but BuildKit cache exports use a custom mediatype. Co-authored-by: Cursor <cursoragent@cursor.com>
BuildKit exports cache with a custom mediatype (application/vnd.buildkit.cacheconfig.v0) that can't be unpacked by standard OCI tools like umoci. This caused errors when pushing cache images to the registry: config blob is not correct mediatype application/vnd.oci.image.config.v1+json: application/vnd.buildkit.cacheconfig.v0 The fix skips the ext4 conversion step for cache/* repos since: 1. Cache images are not runnable containers 2. BuildKit imports them directly from the registry 3. There's no need to unpack or convert them locally Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
The repo parameter passed to triggerConversion includes the Host header
prefix (e.g., "10.102.0.1:8083/cache/global/node"). The previous check
only used HasPrefix("cache/") which would never match.
Now checks for both patterns:
- HasPrefix("cache/") for edge case without host
- Contains("/cache/") for normal case with host prefix
Co-authored-by: Cursor <cursoragent@cursor.com>
sjmiller609
approved these changes
Feb 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two related issues with BuildKit cache handling:
image-manifest=true, BuildKit's registry cache stores layer references pointing to external registries rather than copying actual layer blobsapplication/vnd.buildkit.cacheconfig.v0) that can't be unpacked by standard OCI toolsChanges
Fix 1: Enable proper cache blob storage
lib/builds/builder_agent/main.go- Addedimage-manifest=true,oci-mediatypes=trueto cache export optionslib/builds/cache.go- UpdatedExportCacheArg()helper for consistencylib/builds/cache_test.go- Updated test expectationFix 2: Skip conversion for cache images
lib/registry/registry.go- SkiptriggerConversion()forcache/*repos since they're not runnable containersTest coverage
lib/images/oci_test.go- Unit tests documenting the BuildKit cache mediatype limitationRoot causes
Issue 1: Without
image-manifest=true, BuildKit stores layer references like:Instead of copying the actual blob. Ephemeral BuildKit instances can't resolve these references.
Issue 2: When cache images are pushed, the registry triggers conversion to ext4 format. This calls
umoci.UnpackRootfswhich expects standard OCI config mediatype but BuildKit uses:This caused:
config blob is not correct mediatype application/vnd.oci.image.config.v1+jsonBefore (first tenant deployment)
After
Test plan
go test ./lib/images/... ./lib/registry/... ./lib/builds/...)builder_agentNote
Medium Risk
Touches build caching behavior and registry post-push processing; incorrect cache flags or repo matching could reduce cache effectiveness or unintentionally skip conversion for some images.
Overview
Fixes BuildKit registry cache exports to work reliably in ephemeral builders by adding
image-manifest=true(andoci-mediatypes=true) to cache export args for both global/admin and tenant caches.Updates the shared
CacheKey.ExportCacheArg()helper and its tests accordingly, addslib/images/oci_test.goto document the BuildKit cache config mediatype incompatibility, and changes the registry to skip ext4 conversion for pushedcache/*images so cache pushes no longer fail.Written by Cursor Bugbot for commit 1460df9. This will update automatically on new commits. Configure here.