ci: prebuilt runner images for smoke/SDK/web jobs

.agents/skills/manage-ci/references/current-inventory.md

@@ -86,6 +86,246 @@ workflows remain fixed to GitHub-hosted runners; the PR entrypoints pass no
 repository secrets. The trusted main entrypoint may pass the optional
 `HF_TOKEN` for public-fixture rate-limit resilience.
 
+## Prebuilt runner-image containerization
+
+Some CI jobs run inside a `container:` pinned to a digest from the
+`mesh-llm-runner-images` repo instead of installing tooling per-run with
+`actions/setup-*`. There is no separate sister repo: the `public web`
+backend (baked Chromium/Playwright) lives on `mesh-llm-runner-images` main
+alongside every other family, added by `17283ab` (#20, the `public web`
+backend) and `5ea673b` (#21, the Playwright version assert). The GHCR
+package name
+`ghcr.io/mesh-llm/mesh-llm-cuda-runner` is legacy: it hosts every backend
+family (`public cpu`, `public cuda`, `public rocm`, `public vulkan`,
+`public web`, `self-hosted`), not only CUDA. Each image bakes
+`cargo cmake docker git jq just lld node ninja npm pnpm python rustc sccache`
+(asserted by `verify-runner-image`, see below) plus a Python venv on `PATH`
+(`VIRTUAL_ENV=/opt/mesh-llm/venv`), pinned pnpm/node (`PNPM_HOME`,
+`CARGO_HOME`, `RUSTUP_HOME` baked as ENV so they resolve the same regardless
+of the container's `HOME`), and, for the `public` stage only, runs as
+**root** (`USER root`, never dropped back) rather than `runner` --
+`self-hosted` is the only stage that ends `USER runner`.
+
+Reusable slices/workflows with a `container:` job, and what backs it:
+
+| Workflow | Job(s) | Image family |
+| --- | --- | --- |
+| `ci-{linux}-host-slice.yml`, `ci-linux-runtime-slice.yml`, `ci-linux-product-slice.yml`, `ci-rust-tests-slice.yml`, `ci-quality-slice.yml` (Clippy batches) | matrix-selected | `public cpu` (pre-existing, predates this containerization pass) |
+| `native-sdk-artifact.yml`, `node-sdk-addon-artifact.yml`, `static-abi-artifact.yml`, `swift-sdk-artifact.yml` | producer job | `public cpu` (pre-existing) |
+| `hf-download-smoke.yml`, `scripted-binary-smoke.yml` | their single job | `public cpu`, sha256:8d93de6b... -- unconditional, no bare-metal row |
+| `smoke.yml` | `smoke_tests` | `public cpu` when `inputs.runner != 'gpu-nvidia'`, else uncontainerized (see opt-out below) |
+| `sdk-smoke.yml` | its job | `public cpu` when `inputs.sdk_kind != 'swift'`, else uncontainerized |
+| `ci-ui-artifact-slice.yml` | `ui_artifact` | `public web`, sha256:1c73f0f2... |
+| `ci-web-slice.yml` | `ui_quality`, `ui_e2e`, `website` | `public web` |
+| `website-pages.yml` | `build` | `public web` |
+| `nightly-stability-run.yml` | `stability` | `public web` (bakes node/pnpm the CLI-smoke step needs) |
+| `release.yml` (several CUDA/ROCm/Vulkan build/compose rows) | per-backend `public` digests | pre-existing, unrelated to this containerization work; each row pins its own backend digest via `ci/slices.yml` / job matrix, not a shared convention |
+
+`public cpu` and `public web` are separate image builds (the latter adds
+`PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright`,
+`PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1`, and a stamped
+`/etc/mesh-runner-playwright-version`); do not assume one digest covers both.
+
+### `image: ''` opt-out rows
+
+`smoke.yml`'s `gpu-nvidia` row (the approved uncredentialed self-hosted CUDA
+smoke exception) and `sdk-smoke.yml`'s `swift` row (host-only macOS SDK
+build, `macos-15`, never container-capable) opt out per-run with
+`image: ''` rather than being a separate job, so the rest of the job body
+(steps, `if: job.container.id == ''` gates) stays shared. This is proven to
+actually opt a job out of containerization by two runs on the temporary
+branch-head harness used to validate #1380 (run 32349670919, jobs
+96370649138 `CUDA inference smoke` and 96375145155 `swift SDK Smoke`): no
+`Initialize containers` log group, no `docker create`, and the gated
+`actions/setup-python`/`pnpm/action-setup` steps ran. There is no other
+empty-image job anywhere in this repo's workflow history.
+
+The ternary that selects `image: ''` must put the **non-empty** value in the
+`&&` branch: `cond && url || ''`, never `cond && '' || url`. GitHub Actions
+expressions are JS-style short-circuit and `''` is falsy, so
+`cond && '' || url` always evaluates to `url` regardless of `cond` -- the
+opt-out branch becomes unreachable. `scripts/tests/test_ci_workflow_ternary_contract.py`
+fails any `${{ }}` ternary whose `&&` branch is a falsy literal (`''`, `""`,
+`0`, `false`) across every workflow; it exists specifically because this bug
+class is invisible to `actionlint`.
+
+### `job.container.id == ''` gating
+
+When a job has both a containerized and a bare-metal row (the two rows
+above), `actions/setup-python`, `actions/setup-node`, and
+`pnpm/action-setup` steps are gated `if: job.container.id == ''` rather than
+deleted, because the bare-metal row still needs them -- the image is not
+present there. **Deliberate exception: `actions/setup-java` in
+`sdk-smoke.yml` is never gated.** `verify-runner-image`'s asserted tool list
+has no JDK, so the image provides nothing for it to shadow; gating it would
+break the Kotlin SDK smoke on the containerized row instead of protecting it.
+Jobs with no bare-metal row at all (the `ci-web-slice.yml` / `website-pages.yml`
+/ `ci-ui-artifact-slice.yml` / `nightly-stability-run.yml` set) delete the
+now-redundant setup actions outright instead of gating them -- there is
+nothing for the `if:` to select between.
+
+`npm install --global openai` in `smoke.yml` is **not** gated on
+`job.container.id`, even though `install-core-tools.sh:83` bakes an
+exact-pinned `openai` into the image on `mesh-llm-runner-images` main. The
+`public cpu` digest pinned in `ci/slices.yml` predates that bake (see
+"A pinned digest is a frozen artifact" below), so the containerized row needs
+the install too, and the step runs unconditionally for both it and the
+bare-metal `gpu-nvidia` row. Re-gate it only once the CPU digest is promoted
+past `mesh-llm-runner-images` #20 and that is confirmed from a green run.
+
+Both call sites install `openai@7.5.0`, not floating `openai`. The step runs
+with the full job environment (`HF_TOKEN` included) and npm lifecycle scripts
+inherit it, so an unreviewed upstream release must not be able to execute
+there; `zizmor`'s `adhoc-packages` rule flags the floating form. The version
+deliberately tracks the image's own `ARG OPENAI_NPM_VERSION`
+(`mesh-llm-runner-images` `Dockerfile:25`) so that re-gating the step later is
+a no-op rather than a version swap -- bump both sides together.
+
+### Container jobs default `run:` to `sh`, not `bash`
+
+Jobs with a `container:` block resolve the default `run:` shell to
+`sh -e {0}`, not `bash -e {0}` (bare-metal Linux/macOS runners default to
+bash; this only changes inside a container). Composite actions are
+unaffected -- they declare their own shell. Any `run:` step in a
+containerized job that uses a bashism (`<<<`, `set -o pipefail`, `[[`,
+array assignment, `${v//}`/`${v^^}`/`${v,,}`, `&>`, `source`, `+=(`, ...)
+must declare `shell: bash` explicitly or it fails at runtime with a
+`dash`/`sh` syntax error that `actionlint` cannot catch -- its shellcheck
+integration assumes bash. Two sites hit this in the same PR:
+`ci-web-slice.yml`'s `ui_e2e` preflight (`<<<`) and
+`website-pages.yml`'s `Stage Pages artifact` (`set -euo pipefail`); both now
+declare `shell: bash`.
+
+`$(( ))` arithmetic expansion is **not** on that list and must not be added.
+It is POSIX (Shell Command Language 2.6.4) and `dash` evaluates it correctly;
+flagging it would reject valid `sh` steps and force a spurious `shell: bash`.
+`scripts/tests/test_ci_workflow_container_shell_contract.py` carries the
+pattern list and an inline note saying so.
+
+### Reusable-workflow permission chain
+
+A called reusable workflow may not request a permission scope its caller job
+does not grant; GitHub rejects at run creation with a **zero-job
+`startup_failure`** -- no jobs, no logs, no check run on the commit, and
+`actionlint` cannot see it. Containerizing surfaced this because
+`packages: read` (needed to pull the private GHCR runner images) has to be
+granted at *every* hop, and
+`ci-linux-product-smoke-slice.yml` / `ci-macos-product-smoke-slice.yml` sat at
+`contents: read` between granted parents and requesting children.
+`scripts/tests/test_ci_workflow_permission_contract.py` walks every local
+`uses: ./.github/workflows/X.yml` edge and asserts the caller's effective
+permissions (job-level, else workflow-level) cover what `X.yml` requests.
+
+Two properties make that assertion real rather than decorative, and both were
+absent when the test was first written:
+
+1. **The callee's requested set is the workflow-level block merged with every
+   explicit job-level block.** Five reusable workflows here
+   (`native-sdk-artifact.yml`, `node-sdk-addon-artifact.yml`, `sdk-smoke.yml`,
+   `static-abi-artifact.yml`, `swift-sdk-artifact.yml`) declare permissions
+   only at job level, so reading the workflow-level block alone returns `None`
+   for them and skips their caller edges entirely -- including the
+   `packages: read` edges this test exists to cover.
+2. **Scope levels are compared, not scope names.** `contents: read` does not
+   satisfy a callee's `contents: write`; GitHub rejects that downgrade at run
+   creation exactly like a missing scope. The comparison ranks
+   `none < read < write`, and where a scope is declared in more than one block
+   the strictest level wins. A name-only set comparison silently passes the
+   downgrade.
+
+3. **`read-all`/`write-all` are modelled on the granting side, not skipped.**
+   As a *grant* they are perfectly enumerable -- `write-all` satisfies any
+   request, `read-all` satisfies a `read` request but not a `write` one -- so
+   returning "unknown" and skipping the edge would hide the same
+   run-creation failure. As a *request* they stay opaque: a callee asking
+   `write-all` names no scopes to hold its caller to, and asserting there
+   would be invention rather than checking.
+4. **Both workflow extensions are read.** Globbing `*.yml` alone would skip a
+   `*.yaml` callee entirely; the repo has none today, which is exactly when
+   that gap is cheapest to close.
+
+None of these were breakages -- the repo satisfies the contract at every edge
+under the strict check, and it has no `.yaml` workflows or all-scope grants at
+all. That is the point: a permission test that under-reads its inputs reports
+green for edges it never examined, and each of these was found by tightening
+the test rather than by anything failing.
+
+### `verify-runner-image` preflight
+
+Containerized jobs run `verify-runner-image <environment> <backend> ...`
+(positional args: environment, backend, mesh-llm revision, CUDA series, ROCm
+version, runner-images revision, and -- `public web` only -- expected
+Playwright version, added in `mesh-llm-runner-images`#21) before doing real
+work, asserting `/etc/mesh-runner-*` files match what the job expects rather
+than trusting the digest pin alone. `ci-web-slice.yml`'s `ui_e2e` job
+resolves the installed `@playwright/test` version with
+`pnpm exec playwright --version | head -n1 | awk '{print $NF}'` (guarded by a
+`^[0-9]+\.[0-9]+\.[0-9]+$` shape assertion -- `playwright --version` can share
+stdout with an npm warning) and passes it as the seventh argument; a mismatch
+against the image's own build-time `playwright --version` fails fast instead
+of surfacing as a confusing Playwright/Chromium error deep in the E2E run.
+
+`crates/mesh-llm-ui/package.json`'s `@playwright/test` and
+`mesh-llm-runner-images`' `config/playwright-pin.txt` are now a matched pair
+(both `1.62.1` as of 2026-08-20; re-check the two sources rather than
+trusting this line). Bumping the mesh-llm side alone fails `ui_e2e` on
+**every** PR at this preflight, not just locally. The bump is a four-step
+cross-repo sequence, in order: bump `config/playwright-pin.txt` in
+`mesh-llm-runner-images`, rebuild and promote the `public web` image, re-pin
+the new digest in `ci-web-slice.yml` (and `ci-ui-artifact-slice.yml` /
+`website-pages.yml` / `nightly-stability-run.yml`, which share it), then
+bump `@playwright/test` in `crates/mesh-llm-ui/package.json`.
+
+### `setup-macos-lld` composite
+
+`.github/actions/setup-macos-lld` replaces per-callsite
+`brew install lld` plus a hand-rolled `PATH`/`RUSTFLAGS` export with one
+composite: install lld via brew, resolve `$(brew --prefix lld)/bin`, link a
+real `edition = "2024"` probe binary with `-Clink-arg=-fuse-ld=lld`, then
+export `CARGO_ENCODED_RUSTFLAGS=-Clink-arg=-fuse-ld=lld` and the resolved bin
+directory. `CARGO_ENCODED_RUSTFLAGS` **replaces** any
+`target.<triple>.rustflags` from a checked-in `.cargo/config.toml` rather
+than merging with it -- confirmed safe here because every call site is
+macOS-gated and no call site touches an `android` target (the repo's
+`.cargo/config.toml` android entries carry a `-Wl,-z,max-page-size=16384`
+flag that would otherwise silently stop applying). Seven call sites:
+`ci-platform-checks-slice.yml`, `ci-macos-host-slice.yml`,
+`swift-sdk-artifact.yml`, `native-sdk-artifact.yml`,
+`node-sdk-addon-artifact.yml`, and two in `release.yml`. Only the first
+three were reachable by the temporary branch-head harness used to validate
+PR #1380 (no macOS row in
+`native-sdk-artifact.yml`/`node-sdk-addon-artifact.yml` ran there, and
+`release.yml` only runs on an actual release cut) -- the other four are
+statically cleared (macOS-gated, no android target on any of them) rather
+than proven by a real run. `node-sdk-addon-artifact.yml`'s own
+`Validate macOS x64 cross-linker` step is a deliberate near-duplicate of the
+composite's probe, not dead code: it passes `--target x86_64-apple-darwin`
+where the composite only probes the host target.
+
+### A pinned digest is a frozen artifact
+
+`mesh-llm-runner-images` HEAD says nothing about what is inside the digest a
+workflow pins -- the `public cpu` digest pinned in `ci/slices.yml` was built
+2026-07-22 and does not contain changes merged to that repo afterwards
+(the `smoke.yml` openai bake landed a week later, in #20). Before deleting or
+gating a dependency install on the grounds that "the image bakes it,"
+confirm the capability exists **in the pinned digest**, and confirm it from a
+green run of the job that needs it. `verify-runner-image`'s JSON is the
+cheap probe: `mesh_llm_revision` dates the build, and missing keys (added to
+the asserted object in later `mesh-llm-runner-images` commits) date the
+baked verify script itself.
+
+### Digest promotion
+
+`build-and-push.yml` (in `mesh-llm-runner-images`) runs `stage_families` for
+both `operation=stage` and `operation=promote`; `promote_versioned` reads the
+candidate descriptor artifact from that **same run**, not from an earlier
+stage run. A `promote` dispatch therefore re-stages and promotes its own
+build. Read the digest to pin from the promote job's own `digest=` output
+(e.g. `promoted ghcr.io/... -> sha256:...` in its log) -- never carry forward
+a digest observed from an earlier stage-only run, even one at the same
+source commit.
+
 ## Planner contract
 
 - `scripts/plan-ci.py` is the only routing implementation.

.github/actions/setup-macos-lld/action.yml

@@ -0,0 +1,38 @@
+name: Set up macOS lld linker
+description: >
+  Install and verify Homebrew's lld on a macOS runner. Always probe-links a
+  throwaway crate before any caller trusts the result -- a bad install must
+  fail loudly, not build silently against the wrong (or no) linker.
+  Deliberately uncached: mesh-llm's Actions cache is already over the 10GB
+  per-repo quota (see gh cache list), so a ~1.9GB lld/llvm/z3 Cellar entry
+  here would evict the far more expensive macOS Rust target/ cache for a
+  15-30s bottle-pour saving. Revisit only with a measured number and a
+  quota plan, not an assumption.
+
+runs:
+  using: composite
+  steps:
+    - name: Install lld
+      shell: bash
+      run: brew install lld
+
+    - name: Verify lld links a real binary
+      shell: bash
+      run: |
+        set -euo pipefail
+        lld_bin="$(brew --prefix lld)/bin"
+        test -x "$lld_bin/ld64.lld"
+        probe="$RUNNER_TEMP/mesh-setup-macos-lld-probe"
+        mkdir -p "$probe/src"
+        printf '[package]\nname = "mesh-setup-macos-lld-probe"\nversion = "0.0.0"\nedition = "2024"\n' > "$probe/Cargo.toml"
+        printf 'fn main() {}\n' > "$probe/src/main.rs"
+        PATH="$lld_bin:$PATH" CARGO_ENCODED_RUSTFLAGS='-Clink-arg=-fuse-ld=lld' \
+          cargo rustc --quiet --manifest-path "$probe/Cargo.toml"
+        echo "$lld_bin" >> "$GITHUB_PATH"
+        # Job-global via GITHUB_ENV, and cargo prefers CARGO_ENCODED_RUSTFLAGS
+        # over RUSTFLAGS -- any later step in the same job that sets RUSTFLAGS
+        # is silently ignored, and any macOS job that cross-builds an android
+        # target loses .cargo/config.toml's -Wl,-z,max-page-size=16384 for
+        # that job. Verified safe today: no other RUSTFLAGS setter and no
+        # android targets on any of this action's macOS call sites.
+        echo 'CARGO_ENCODED_RUSTFLAGS=-Clink-arg=-fuse-ld=lld' >> "$GITHUB_ENV"

.github/workflows/ci-linux-host-slice.yml

@@ -120,7 +120,7 @@ jobs:
           ref: ${{ inputs.source_sha || github.sha }}
           persist-credentials: false
       - name: Verify prebuilt host environment
-        run: verify-runner-image public
+        run: verify-runner-image public cpu
       - uses: ./.github/actions/configure-sccache-gha
         with:
           allow_depot_remote_cache: ${{ needs.runner_policy.outputs.allow_depot_remote_cache }}

.github/workflows/ci-linux-product-slice.yml

@@ -98,7 +98,7 @@ jobs:
           ref: ${{ inputs.source_sha || github.sha }}
           persist-credentials: false
       - name: Verify product composition environment
-        run: verify-runner-image public
+        run: verify-runner-image public cpu
       - uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
         with:
           name: ci-host-linux-${{ matrix.runtime.architecture }}

.github/workflows/ci-linux-product-smoke-slice.yml

@@ -26,6 +26,7 @@ on:
 
 permissions:
   contents: read
+  packages: read
 
 jobs:
   core:

.github/workflows/ci-macos-host-slice.yml

@@ -115,7 +115,7 @@ jobs:
           shared-key: ci-host-macos-${{ inputs.profile }}
           save-if: ${{ github.ref == 'refs/heads/main' && github.event.inputs.original_event_name != 'pull_request' && github.event.inputs.original_event_name != 'pull_request_target' }}
       - name: Install host linker
-        run: brew install lld
+        uses: ./.github/actions/setup-macos-lld
       - name: Download immutable UI distribution
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
         with:

.github/workflows/ci-macos-product-smoke-slice.yml

@@ -30,6 +30,7 @@ on:
 
 permissions:
   contents: read
+  packages: read
 
 jobs:
   metal_model_load:

.github/workflows/ci-macos-runtime-slice.yml

@@ -98,8 +98,6 @@ jobs:
         with:
           ref: ${{ inputs.source_sha || github.sha }}
           persist-credentials: false
-      - name: Install Metal runtime dependencies
-        run: brew install cmake ninja jq
       - uses: ./.github/actions/resolve-native-toolchain-epoch
         with:
           include_tool_versions: "true"

.github/workflows/ci-platform-checks-slice.yml

@@ -114,14 +114,7 @@ jobs:
       - uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 # stable 2026-07-16
       - name: Configure macOS linker
         if: ${{ matrix.check.platform == 'macos' }}
-        run: |
-          brew install lld
-          mkdir -p .cargo
-          lld_prefix="$(brew --prefix lld)"
-          cat > .cargo/config.toml <<EOF
-          [target.aarch64-apple-darwin]
-          rustflags = ["-C", "link-arg=-fuse-ld=$lld_prefix/bin/ld64.lld"]
-          EOF
+        uses: ./.github/actions/setup-macos-lld
       - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
         if: ${{ matrix.check.platform == 'windows' }}
         with:
@@ -156,7 +149,7 @@ jobs:
         run: cargo test --locked -p mesh-llm-log-store --lib sqlite_root_database_and_sidecars_have_only_current_user_acl
       - name: Install macOS unit-test dependencies
         if: ${{ matrix.check.kind == 'unit' }}
-        run: brew install cmake ninja jq lld
+        run: brew install cmake ninja jq
       - name: Resolve macOS unit-test toolchain epoch
         if: ${{ matrix.check.kind == 'unit' }}
         id: native_toolchain