RFC: Pre-seed BuildKit layer store to reduce build times

[hiroTamada]

Summary

• Proposes pre-seeding the builder VM rootfs with extracted base image layers to eliminate the ~10.6s decompression bottleneck on every code-change deployment
• Evaluates three approaches: pre-seed at build time, persistent volumes per tenant, shared read-only cache
• Recommends pre-seeding at build time as the simplest option with ~37% build time reduction

Problem

Every code-change deployment takes ~27s of build time. The single largest cost (~10.6s) is decompressing and extracting base image layers inside the ephemeral builder VM. This happens because:

1. Builder VMs start with an empty BuildKit content store
2. Any cache miss (code changed) requires BuildKit to reconstruct the base image filesystem
3. The registry stores compressed layers, but BuildKit needs uncompressed filesystem trees

Expected Impact

Scenario	Current	Proposed
Code change deploy	~27s build	~17s build (-37%)
Total deploy time	~50s	~40s

Open Questions

1. Should we pre-seed multiple base images or just nodejs?
2. Acceptable builder image size increase? (~100-150MB per base image)
3. Should warm-up be in CI/CD or manual when base images change?

🤖 Generated with Claude Code

[sjmiller609]

9-10MB/s decompression seems like something might be wrong here 🤔

Normal single-core bound gzip is 80-150MB/s. but Lz4 fast decompress can be 2k-4k MB/s

So off the bat here I am suspicious of this finding indicating something is messed up in the configuration in comparison to just expected slow extract

[sjmiller609]

I'm not sure but I'm curious if alternate compression algos are supported on buildkit and / or if that would work in the current architecture. Like can we save as zstd isntead of gzip? can we support parallel extraction instead of single core bound?

[sjmiller609]

It's probably better to first figure out why it's slow than try to swap algos. Then swap algos after it's already going as fast as normal gzip should be

[sjmiller609]

I kind of feel like pre-seed approach is a way to workaround some bottleneck that shouldn't exist. I am suspicious that this extraction speed of ~10MB/s is over 10x slower than it ought to be even given single-core gzip extraction. It may be advisable to better understand why it's slow before jumping into a pre-caching / warm pooling kind of pattern to work around it being too slow.

it seems like 100MB change should take about 1 or 2s to extract so it just doesn't seem to add up and implies some configuration or VM performance issue