Skip to content

feat(containers): build, gate and publish GHCR images (ENG-RELEASE-CONTAINERS) - #307

Merged
localai-bot merged 1 commit into
mainfrom
row/ENG-RELEASE-CONTAINERS
Aug 11, 2026
Merged

feat(containers): build, gate and publish GHCR images (ENG-RELEASE-CONTAINERS)#307
localai-bot merged 1 commit into
mainfrom
row/ENG-RELEASE-CONTAINERS

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Implements ENG-RELEASE-CONTAINERS — spike spec plus W1-W5 and W7. Closes #170 (image lanes), closes #312 (the bug the gate found).

The cpu lane is built and gated end to end on a real image. Nothing is published: the GHCR package does not exist, and no lane has matching-hardware runtime evidence — that is W6 and it stays open.

What is here

One docker/Dockerfile, three lane targets. Every builder stage calls the same scripts/build-*-release.sh the release workflow calls, so the ten-SM gencode list, the CPU tier gates and the staged layout are written down exactly once. Those scripts already end in validate-release-archive.py, so an image build inherits the extracted-archive audit rather than reimplementing it.

Runtime is ubuntu:24.04 pinned by digest for every lane; the cuda lane copies only the CUDA runtime libraries it links, and the driver stays on the host. ffmpeg ships in all three lanes — a deliberate, argued divergence from the archive contract's no-vendor rule, because an archive is extracted onto a host with a PATH to inherit and a container has none.

Three new gates, each with a red-first mutation suite:

Gate Enforces Suite
check-container-matrix.py matrix ↔ Dockerfile agree on lanes, tags, digest-pinned bases, driver boundary 31/31
check-container-workflow.py no registry write in the PR-facing job, OIDC only in attest, publish tag-gated, validation before push 22/22
validate-container-image.py config, layout, and — given a model — a real boot run on a real image

Measured

x86_64, Docker 29.1.2, booted on opt-125m-bf16-st:

  • 783 MB cpu image; config + layout + boot all green
  • /health 200, /version 200, the image's own declared healthcheck passing inside the container, clean SIGTERM

The gate found a real bug on its first run (#312)

This is why the smoke boots the server instead of running --help.

vllm-server installed no SIGTERM handler. As container PID 1 the kernel does not apply default signal dispositions, so the signal was ignored outright: docker stop waited its full 30 s grace, then SIGKILL — exit 137. Every rolling update, compose down and systemd restart was hard-killing the server and dropping in-flight work. It was invisible outside a container because a non-PID-1 process dies on the default disposition anyway.

Fixed with a self-pipe handler routing SIGTERM/SIGINT into the same server.stop() the existing VT_BENCH_PROFILE_CONTROL FIFO shutdown already used, installed at all three listen() sites.

RED: exit 137 after 30 s → GREEN: exit 0 in 0.25 s.

Also fixed, and silent until now

.dockerignore's **/build*/ matched files as well as directories — Docker does not honour a trailing slash as directories-only the way .gitignore does — so scripts/build-*-release.sh were excluded from the build context and the image build failed with not found after appearing to copy the whole tree. The builders also needed file and binutils, without which the inherited archive validator failed after a full compile. -j 2 was hardcoded in both release scripts and is now a JOBS parameter.

Not established, and not claimed

  • Nothing is pushed to GHCR; the package does not exist.
  • cuda and vulkan are gated statically but have never been built here.
  • Both arm64 legs are unbuilt, so SBSA-vs-Tegra (Thor sm_110, Orin sm_87) is untouched.
  • Hosted CI has no model, so it runs config and layout only and reports the absence of runtime evidence rather than implying it.

Records

Row → ACTIVE with anchors and a claim. docs/STATUS.md lands 1 byte under its ratchet, which is lowered in this change with the reason attached. FEATURES gains container and graceful-shutdown rows; BENCHMARKS gains the image size and the shutdown number; USAGE documents pulling, running and validating an image.

scripts/agent-preflight.sh and scripts/agent-ready.py are green on the merged tree, and the image was rebuilt and re-gated after merging main, since main touched server_main.cpp.

🤖 Generated with Claude Code

@localai-bot localai-bot changed the title spec(containers): scope GHCR container images (ENG-RELEASE-CONTAINERS) feat(containers): build, gate and publish GHCR images (ENG-RELEASE-CONTAINERS) Aug 10, 2026
@localai-bot

Copy link
Copy Markdown
Collaborator Author

The two red sanitizer lanes are pre-existing on main, not from this PR

Verified by isolation rather than assumed. Built and ran the failing test under TSan on pristine origin/main d06a3792, with none of this branch applied:

WARNING: ThreadSanitizer: data race
    #8 PromRegistry::SeriesFor(...)  src/vllm/v1/metrics/prometheus.cpp:134
    #9 PromRegistry::SetGauge(...)   src/vllm/v1/metrics/prometheus.cpp:165

It reproduces there. Filed as #330 with the root cause: SeriesFor returns a Series& into a std::vector<Series> and then push_backs into that same vector, so every previously returned reference dangles — and PromRegistry carries no mutex or atomic at all while /metrics is now served concurrently.

Supporting evidence that it is not this branch:

  • This PR's spec-only commit c3db7804 — no C++ whatsoever — passed both sanitizer lanes. The failures appear only after catch-up merges that pulled in 9c55e3df (/metrics is dead on the production server: AsyncLLM never folds IterationStats into the logger #277).
  • The entire C++ diff here is server_main.cpp (+86, the SIGTERM handler). The ASan lane fails six tests, three of them loaders (test_load_direct_upload, test_laguna_nvfp4_loader, test_llama_embedding_fold) that a signal handler cannot reach.

Per the protocol a bug found while doing something else gets its own issue rather than a silent fix, and the metrics row owns that code — so #330 is filed and not repaired here.

The container-specific checks are green

containers.yml ran end to end on this PR: plan ✅, verify (cpu, linux/amd64) ✅, verify (vulkan, linux/amd64) ✅ — the vulkan lane's first build anywhere — with publish/manifest/attest/promote correctly skipped on a non-tag.

localai-bot pushed a commit that referenced this pull request Aug 11, 2026
…sanitizer red

FOLLOWING_AGENTS_PROTOCOL

containers.yml ran end to end on PR #307: plan green, verify (cpu, amd64) and
verify (vulkan, amd64) green -- the vulkan lane built for the first time
anywhere -- with publish/manifest/attest/promote correctly skipped on a
non-tag. The records said both accelerator lanes had never been built; that is
now true only of cuda, and the spec and matrix say so.

CI has no model, so those runs cover config and layout and REPORT the absence
of runtime evidence rather than implying it.

Separately: the two red sanitizer lanes are NOT from this branch. Reproduced on
pristine origin/main d06a379 -- PromRegistry::SeriesFor returns a Series& into
a std::vector it then push_backs into, so every prior reference dangles, and
the registry holds no mutex while /metrics is served concurrently. Filed as
#330 and left to the metrics row rather than silently repaired here. This
branch is C++-clean apart from the SIGTERM handler in server_main.cpp, and its
spec-only commit c3db780 passed both lanes before those merges landed.

Issues: #170, #330

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]
…NTAINERS)

FOLLOWING_AGENTS_PROTOCOL

Spike spec plus W1-W5 and W7 of ENG-RELEASE-CONTAINERS. The cpu lane is built
and gated end to end on a real image; vulkan builds and validates on hosted CI;
cuda is implemented and gated statically but has never been built. NOTHING is
published: the GHCR package does not exist and no lane has matching-hardware
runtime evidence (W6).

One docker/Dockerfile with three lane targets. Every builder stage calls the
SAME scripts/build-*-release.sh the release workflow calls, so the ten-SM
gencode list, the CPU tier gates and the staged layout are written down once.
Those scripts already end in validate-release-archive.py, so an image build
inherits the extracted-archive audit rather than reimplementing it -- which is
why the builders carry file/binutils, qemu-user and a SHA256-pinned Intel SDE,
and why the vulkan builder needs a software ICD: each is a real requirement of
the gate being inherited. `-j 2` was hardcoded in both release scripts and is
now a JOBS parameter.

New gates, each with a red-first mutation suite:
- check-container-matrix.py keeps release/container-matrix.json and the
  Dockerfile agreeing on lanes, tags, digest-pinned bases and the driver
  boundary (31/31). It scans RUN/COPY/ADD only, so the image can still DOCUMENT
  in a comment and a label that libcuda.so.1 comes from the host.
- check-container-workflow.py holds containers.yml to its stages: no registry
  write in the job a pull request runs, OIDC only in attest, publish tag-gated,
  and validation BEFORE the push (28/28, ordering included).
- validate-container-image.py audits config and layout and, given a model,
  boots the server and requires /health, /version, the image's own declared
  healthcheck, and a clean SIGTERM.

MEASURED (x86_64, Docker 29.1.2, opt-125m-bf16-st): 783 MB cpu image, config +
layout + boot all green. Hosted CI: plan, verify(cpu, amd64) and
verify(vulkan, amd64) green, publish stages correctly skipped on a non-tag.

The boot gate found a real bug on its first run, which is why it boots the
server instead of running --help (#312, fixed here): vllm-server installed no
SIGTERM handler, and as container PID 1 the kernel does not apply default
signal dispositions, so the signal was IGNORED -- docker stop waited its full
30 s grace and then SIGKILLed, exit 137. A self-pipe handler now routes SIGTERM
and SIGINT into the same server.stop() the VT_BENCH_PROFILE_CONTROL FIFO
shutdown already used, at all three listen() sites. RED 137 after 30 s -> GREEN
exit 0 in 0.25 s. Every rolling update and compose down was hard-killing the
server; it was invisible outside a container because a non-PID-1 process dies
on the default disposition anyway.

Three further bugs, none of which could fail anywhere but in a real release:
- .dockerignore's `**/build*/` matched FILES as well as directories, because
  Docker does not honour a trailing slash the way .gitignore does, so
  scripts/build-*-release.sh were excluded from the build context.
- `docker push --quiet` prints the image REFERENCE, not the digest, so the
  manifest stage would have joined garbage.
- The immutable-tag guard ran BEFORE login, where imagetools inspect fails for
  lack of auth -- read as "tag does not exist", a fail-open that would have
  overwritten an immutable version tag.

Pull requests build the lanes that opt in (cpu and vulkan on amd64) rather than
all six, because a ten-SM fat CUDA image does not fit a hosted runner budget per
push. That is a cost decision and NOT a publication hole: publish consumes the
full release matrix and revalidates each lane immediately before pushing it, and
the guard enforces that split.

scripts/check-pr-size.py FAILS CLOSED on unclassified paths, so docker/Dockerfile,
docker/healthcheck.sh and release/container-matrix.json had to be classified
before this change could pass the project's own size gate. Both checker pins
touched here gain a case locking why the pin is justified, not just its value.

The two red sanitizer lanes are NOT from this change: reproduced on pristine
origin/main, root-caused to PromRegistry::SeriesFor returning a Series& into a
std::vector it then push_backs into, and filed as #330 for the metrics row.

Issues: #170, #312, #330

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]
@localai-bot
localai-bot force-pushed the row/ENG-RELEASE-CONTAINERS branch from 87d78aa to 88b740c Compare August 11, 2026 06:54
@localai-bot
localai-bot merged commit b698871 into main Aug 11, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants