diff --git a/.agents/NOW.md b/.agents/NOW.md index 201e08c6..67e02286 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -28,7 +28,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`. | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed (contributor, authorship-preserved); W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) | | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) | -| Release | SPIKE; 30/30 | #129 | +| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** | #141; artifacts pending | | Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139 IN; **ROW 6 LANDED (#137): embeddings LIVE — `LlamaModel` arch, PoolingRunner in the step, `vllm_embed` v15, `/v1/embeddings`, fold gate 4/4-231, 9 kills** | Real-checkpoint oracle cosine residual | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 70aa7322..ad5fb84c 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -16177,3 +16177,29 @@ the ~30 flushes/token. The declines name exactly three ops: `kCausalConv1dFwd` (op 5, prefill only), `kRopeCosSinCache` (op 66) and `kAttnQkNormRopeGate` (op 67). The last two run per full-attention layer, 16 per token, and are the next lever; they are kernel work, not plumbing. + +## Rolled out of the scoreboard on 2026-08-09 + +Moved verbatim from `docs/BENCHMARKS.md` by `scripts/roll-benchmark-record.py`. Nothing edited or deleted. + +## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700) + +| Path | Warm tok/s | Notes | +|------|------------|--------| +| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill | +| server `/v1/completions` | ~38 | exclusive | +| server `/v1/chat` thinking off | ~32 | after expert cache | +| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack | + +## Rolled out of the status page on 2026-08-09 + +Moved verbatim from `docs/STATUS.md`; the keyed capability row remains the +public status surface. + +## 2026-08-08 — Gemma4 ROCm fused helpers via portable vt:: seam (#154) + +Model files (`gemma4.cpp`, `gemma4_moe.cpp`) no longer call `vt::rocm::*` directly. +Fused paths go through `include/vt/fused_ops.h` (`vt::RmsNormPlusAdd`, +`DualRmsNormPlusRes`, `GeluMulSeparate`, `MatmulBTAlphaBeta`, `MatmulBTFp8Channel`, +`ExpertGeGLUBf16TopKM1`). ROCm fast path under `VLLM_CPP_HIP`; non-HIP stubs for +peer/pin/resident upload. `check-device-leakage` holds baseline. diff --git a/.agents/coordination.md b/.agents/coordination.md index f61a85c4..6e1386c1 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,23 +118,19 @@ without the selected contention proof for their entire run are discarded. ## Active claims -**Server binary release-matrix spike (`ENG-RELEASE-BINARIES`, 2026-08-07, -`CLAIM-ENG-RELEASE-BINARIES-SPIKE`).** Codex (GPT-5), isolated worktree +**Server binary release manifest W5 (`ENG-RELEASE-BINARIES`, 2026-08-08, +`CLAIM-ENG-RELEASE-BINARIES-W5`).** Codex (GPT-5), isolated worktree `/home/mudler/_git/vllm.cpp-release-binaries`, branch -`row/ENG-RELEASE-BINARIES`, base `origin/main` `9cb799cb`. Records/spec-only -claim for issue #117 and the user-approved hybrid stable/preview downloadable -server-bundle matrix, including the fully-static feasibility boundary. Owns -only NEW `.agents/specs/release-binary-matrix.md`, the new -`ENG-RELEASE-BINARIES` engine-matrix row, its single roadmap portfolio link, -this claim, the required current-state checkpoint in `docs/STATUS.md` and -`docs/BENCHMARKS.md`, the engine inventory count pin in -`scripts/check-agent-record.py`, `.agents/NOW.md`, and the append-only -`.agents/state.md`. -No release workflow, CMake/package implementation, source, test, model, kernel, -generated fixture, download, build, GPU, or service change is in scope. Draft -PR [#129](https://github.com/mudler/vllm.cpp/pull/129) is the claim. The spike -contract incorporates the user-reviewed fat-CUDA/adaptive-CPU primary-artifact -revision and awaits fresh review; the row stays `SPIKE`. +`row/ENG-RELEASE-BINARIES`, base `origin/main` `b38f78a7`. PR #129 merged the +accepted release matrix; the current claim is W5 only: a versioned manifest +schema, deterministic generator/validator, canonical fixtures, mutation tests, +and the narrow release-contract/record surfaces those tests require. No W1-W4 +CUDA/CPU implementation, W6 archive/install target, release workflow, +published artifact, download, GPU, model, kernel, or service change is in +scope. Draft PR #141 is the claim. W5 is implemented and green at 19/19 after +the fresh review's ten production-removal mutations were killed; the row is +`ACTIVE`, while every archive, staged smoke, runtime, correctness, performance, +install/package and publication claim remains pending. **CPU grouped keep-quant GEMM activation-dtype P0 (`QUANT-GGUF-CIQ-GEMM`, 2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`).** Claude Code @@ -1413,7 +1409,7 @@ table, tests, CMake. Details in the state-log entry of the same date. | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| -| `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | `ENG-RELEASE-BINARIES` | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; records/spec only, no build/GPU/download/service work | `row/ENG-RELEASE-BINARIES`, base `9cb799cb`, draft PR #129 | Accepted release-bundle spike plus its matrix/roadmap/coordination/state/NOW and public-doc lifecycle checkpoint; no CMake, workflow, source, test, or artifact implementation | `ACTIVE` | 2026-08-07 — user-reviewed revision complete: primary fat CUDA + adaptive CPU per host ABI, optional per-SM diagnostics; row stays `SPIKE`; awaiting fresh review | +| `CLAIM-ENG-RELEASE-BINARIES-W5` | `ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; CPU/build-time metadata tooling only, no GPU/download/service work | `row/ENG-RELEASE-BINARIES`, base `b38f78a7`, claim commit `29107d0b`, draft PR #141 | W5 implemented: versioned manifest schema + deterministic generator/validator, canonical synthetic CPU/CUDA fixtures, fail-closed mutation tests, release checker/registration and required record/doc checkpoints. Excludes W1-W4, W6-W13, archives, install/package/publish workflows and runtime artifacts | `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; no archive or real runtime/correctness/performance evidence | | `CLAIM-KIMI-LINEAR-W0` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — dedicated W0 spike) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a771cc029e6843be0`; CPU-only, records-only — NO build, NO GPU, NO download (two GPU jobs queued ahead; the W0 GPU golden capture is a SEPARATE later step) | branch `worktree-agent-a771cc029e6843be0`, base `origin/main` HEAD `10dd23ee` | The FULL dedicated W0 spike for `KimiLinearForCausalLM` (Kimi-Linear-48B-A3B) per the spike-first protocol, so W1 implementation can start immediately. Owns ONLY: NEW `.agents/specs/kimi-linear.md`, the `MODEL-TEXT-kimi-linear-*` matrix row (Spike link → the dedicated spec + note; STAYS `SPIKE` because it is actively claimed — protocol forbids a claimed row from being `READY`; checklist mark stays `📋`), this claim row + the narrative block below, the roadmap breadth note, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, the `.agents/NOW.md` live-claim row + stamp, and one `.agents/state.md` entry. **NON-COLLISION:** records-only — touches NO model/kernel/registry/loader source, NO CMake, NO test. Co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` (MLA half) + `CLAIM-KDA-KERNEL` (KDA host refs), exactly as the K3 row is co-owned by `CLAIM-KIMI-K3-SCOPE`/`CLAIM-KIMI-K3-W2-W5`. | `DONE` | 2026-08-05 — **W0 DEDICATED SPIKE LANDED (records-only, NOT pushed).** Authoritative `config.json` fetched (27 layers = 20 KDA + 7 NoPE-MLA; 256e/top-8/1-shared sigmoid `noaux_tc` `routed_scaling=2.446` `first_k_dense_replace=1`; `num_nextn_predict_layers=0` ⇒ no MTP); reuse-vs-new map with our `file:line`; HW-fit (FITS one GB10, 91.5 GiB / 0.77× pool — real e2e SACRED gate reachable); W0-W7 breakdown + GPU golden-capture recipe. NET-NEW = KDA device kernel (host refs landed, `CLAIM-KDA-KERNEL`), NoPE-MLA branch, hybrid schedule/het-KV, loader name-map. Record checkers green (`check-model-checklist`, `check-agent-record`, `check-doc-checkpoint`, `check-now-current`). | | `CLAIM-KIMI-LINEAR-W1` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W1 registry/config/loader scaffolding; forward REFUSES-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a11ae231a71e8694b`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release); NO GPU, NO download — additive TU + one REGISTER line, forward is a `VT_CHECK(false)` stub | branch `worktree-agent-a11ae231a71e8694b`, base `origin/main` HEAD `053116df` | Kimi-Linear-48B-A3B W1 — registry + config + loader + KV-spec scaffolding so the W3-W6 forward can start. Owns ONLY: NEW `include/vllm/model_executor/models/kimi_linear.h`, NEW `src/vllm/model_executor/models/{kimi_linear_registry,kimi_linear_weights,kimi_linear}.cpp`, NEW `tests/vllm/models/test_kimi_linear_scaffold.cpp`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `KimiLinearForCausalLM` sorted-set + error-message + model-property inserts in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the forward refuses, rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` refuse stub so NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W1 CPU SCAFFOLDING LANDED (foreground, NOT pushed).** Additive registry (`kimi_linear`/`KimiLinearForCausalLM`, `is_hybrid`, `supports_multimodal=false`) + `ParseKimiLinearParams` (20 KDA + 7 NoPE-MLA schedule, MLA 512/128/64/128 q_lora-null, 256e/top-8/1-shared sigmoid `noaux_tc` 2.446, `first_k_dense_replace=1`; asserts `mla_use_nope`/`q_lora==null`) + `EnumerateKimiLinearTensors` name-map VERIFIED vs the real HF safetensors index (MoE = `block_sparse_moe.*`, the correction of the DERIVED K3 `mlp.*`) + loader THROWS BY NAME on missing/mis-shaped tensors + het KV spec (MLA latent-576 + KDA mamba `12288×3`/`32×128×128`) + REFUSE-by-name forward. `test_kimi_linear_scaffold` **9/9·83** + `test_model_registry` **24/24** + `test_kimi_k3_scaffold` 6/6 unchanged; clean CPU Release build; `check-fusion-consistency`/`check-runner-routing-consistency` green (refuse stub skipped); record checkers rc=0. RESIDUAL (W3-W6): KDA device kernel + NoPE-MLA route + sigmoid-noaux MoE + het-KV born-on-runner forward + the e2e SACRED gate (spec §8 recipe). | | `CLAIM-KIMI-LINEAR-W2` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W2-W6 CPU REFERENCE forward; the DEVICE runner forward stays refuse-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-aec13af094cd4692f`; CPU-only `build` (`-DVLLM_CPP_CUDA=OFF` RelWithDebInfo); NO GPU, NO download — additive forward TU + a host-materialization pass in the existing loader + additive test | branch `laguna-s21-w7-speed-profile` (worktree), base `origin/main` HEAD `0988db48` | Kimi-Linear-48B-A3B W2-W6 — the CPU REFERENCE forward, per-op gated, replacing the refuse-by-name host `KimiLinearModel::Forward`. Owns ONLY: NEW `src/vllm/model_executor/models/kimi_linear_forward.cpp`, NEW `tests/vllm/models/test_kimi_linear_forward.cpp`, its two CMake lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), ADDITIVE host-weight structs + per-op decls in `include/vllm/model_executor/models/kimi_linear.h`, the host-materialization pass in `src/vllm/model_executor/models/kimi_linear_weights.cpp`, the device-refuse doc/rename in `src/vllm/model_executor/models/kimi_linear.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the DEVICE forward refuses; rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + additive header structs; the host `Forward` becomes real but `ForwardDevice` (the DEFAULT gather_logits runner path) STAYS `VT_CHECK(false)` so the born-on-runner CI guard skips it (NO allowlist) and NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0/W1`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W2-W6 CPU REFERENCE FORWARD LANDED (foreground, NOT pushed).** Real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives: KDA layer (`vllm::kimi_kda` refs + the gated-delta recurrence `fused_recurrent.py:122-149`), NoPE-MLA (materialized-MHA reference, scaling `qk_head**-0.5`, no RoPE), sigmoid `noaux_tc` MoE (bias-select/unbiased-weight, renormalize, `routed_scaling=2.446`) + shared expert, dense layer-0 SwiGLU; loader now MATERIALIZES the host float weights (bf16/f32→f32). `test_kimi_linear_forward` **6/6·246** (KDA==kimi_kda composition; NoPE-MLA==materialized-MHA ref; router+block==hand-computed; loader materializes; whole 2-layer forward finite/coherent; greedy-decodes 5 tokens, context advances) + `test_kimi_linear_scaffold` 9/9·83 + `test_kimi_kda` 14/14 UNCHANGED; clean CPU build; `check-fusion-consistency`/`check-runner-routing-consistency` green (DEVICE forward refuse-skipped, no allowlist). RESIDUAL: the DEVICE born-on-runner forward + the W0/W7 e2e SACRED token golden on GB10 (spec §8 recipe) + speed. | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 25aef291..032e5438 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 11 | -| Serving, API, CLI, library | 26 | 10 | 2 | 1 | 0 | 5 | 2 | 1 | 5 | +| Serving, API, CLI, library | 26 | 10 | 2 | 0 | 0 | 6 | 2 | 1 | 5 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 3 | 3 | 0 | 0 | 0 | 1 | 1 | 1 | -| **Total** | **144** | **36** | **16** | **4** | **7** | **21** | **8** | **9** | **42** | +| **Total** | **144** | **36** | **16** | **3** | **7** | **22** | **8** | **9** | **42** | ## Engine core and scheduling @@ -207,7 +207,7 @@ claims it. | `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29`; production queue `vllm/v1/engine/core.py:200-231,622-669` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; production `AsyncLLM` benchmark frontend + auditable scheduler depth `examples/bench/bench_core.h:426,495,595`; `examples/bench/main.cpp:51` | server help contract `examples/CMakeLists.txt:34`; production-frontend and metric assertions `tests/examples/test_bench.cpp:18,29-32,61,81,97` | [CLI/serve/benchmark spike](specs/cli-serve-bench.md) | `PARTIAL` | - | | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp:265`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ANCHOR-BACKFILL` | CLAIM-SERVE-GATE-1 | | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - | -| `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | server target only `examples/CMakeLists.txt:54-64`; CPU per-TU/runtime-dispatch baseline `CMakeLists.txt:870-890`, `src/vt/cpu/cpu_matmul_elem.cpp:553-612`, `src/vt/cpu/cpu_quant_dot_arm.cpp:39-77`; cross-family CUDA fat/per-source-gencode and multi-SM AOT gaps remain; no install/archive/publish implementation | help smoke only `examples/CMakeLists.txt:59-63`; issue `#117`; user-reviewed fat-CUDA/adaptive-CPU matrix and gates in [release-binary-matrix.md](specs/release-binary-matrix.md) | [release-binary-matrix.md](specs/release-binary-matrix.md) | `SPIKE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | +| `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | W5 schema `release/manifest-v1.schema.json:1`, deterministic build-time generator/validator `scripts/release_manifest.py:21`, and CPU/CUDA contract fixtures implemented; CPU tier kernel-family/bit/OS-probe inventories are exact and schema const/enum comparisons are JSON-type-strict; server target remains `examples/CMakeLists.txt:54-64`; cross-family CUDA fat/per-source-gencode, multi-SM AOT, CPU forced-tier, install/archive/publish implementation and all real release evidence remain pending | W5 contract/mutation suite `tests/scripts/test_release_manifest.py:260` 19/19 plus ten fresh-review production removals killed independently; existing accepted release-contract suite 30/30; fixtures are synthetic and no archive, staged smoke, runtime, correctness or performance evidence exists | [release-binary-matrix.md](specs/release-binary-matrix.md) | `ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | no Dockerfile for the server exists; the only container asset is the arm64 CPU bench cross-check `docker/Dockerfile.arm64`, which builds `vllm-bench` and is not a release artifact; staging tree, install rule and publish workflow are owed by [`ENG-RELEASE-BINARIES`](engine-matrix.md) and unimplemented | issue `#170`; no image or registry evidence | `planned: specs/container-images.md` | `INVENTORIED` | - | | `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-CLI-CHAT-SPIKE` | | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md). **`/v1/embeddings` LIVE 2026-08-08 (ARCH-ONE-SURFACE ROW 6, `CLAIM-EMBEDDINGS-ONE-SURFACE`):** task-conditional registration (embed/api_router.py:22-28 mirror; the route exists ONLY on a pooling-model server, and the generate routes do not — both directions socket-404-pinned), OpenAI request/response shape (string-or-array input; `dimensions`/base64/token-arrays are named-residual 400s), handler drives the ONE engine path (`LoadedEngine -> LLMEngine::embed -> registry forward -> PoolingRunner`) — the same path `vllm_embed` (ABI v15) drives. RESIDUALS: `/pooling`, `/score`, `/rerank`, `/classify` (need a classify arch) | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/embed/protocol.py:34,173-185`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | `src/vllm/entrypoints/openai/api_server.cpp` `handle_embeddings` + the `if (embedder_)` route gate; `examples/server/main.cpp` pooling task dispatch | `tests/vllm/entrypoints/openai/test_api_server.cpp` embeddings section (dispatch shape + socket smoke + BOTH-direction 404 pins) | [embeddings-one-surface.md](specs/embeddings-one-surface.md) | `ACTIVE` | `CLAIM-EMBEDDINGS-ONE-SURFACE` | diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 2f9a0db2..e8c5f98f 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -932,3 +932,5 @@ Columns: | 2026-08-06 (`KERNEL-CPU-A76-Q8-DOT` R4-R5; `CLAIM-KERNEL-CPU-A76-Q8-DOT`; physical RPi5 Cortex-A76; closing commit: this checkpoint) | Adds an exact-order ACLE SDOT control and an original AAPCS64 two-block Q8_0×Q8_0 leaf behind Linux DotProd/MIDR dispatch. `auto` selects assembly only on Cortex-A76+DotProd; x86, non-DotProd and other Arm CPUs retain portable dispatch. Explicit `portable`/`sdot`/`a76-asm` same-binary controls remain. ARM64 builds/tests locally through buildx/QEMU; Pi is execution-only. | The integer structure is informed by llama.cpp `237ad9b96` `ggml/src/ggml-cpu/arch/arm/quants.c:1076-1160`, while this port deliberately retains the local portable function's per-block f32 reduction order. vLLM `555967922` supplies Qwen3.5 semantics, not a corresponding CPU microkernel. Local anchors: `src/vt/cpu/cpu_quant_dot_{sdot.cpp,a76.S}`, Q8 dispatch in `cpu_quant_dot.cpp`, direct tests in `tests/vt/test_ops_quant_dot.cpp`, and [immutable evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). | **PASS for the compiler-gap/component gate; row `GATING`, `benchmark_binding=true`.** Final binaries `9eb57cf...`/`a94dad30...`; QEMU focused suite 20/20, 150258 assertions; physical-Pi checksums exact. Assembly vs compiler SDOT wall/cycles/instructions: M1/T1 +3.66%/+3.17%/+10.10%, M128/T1 +5.08%/+4.61%/+10.24%, M128/T4 +3.69%/+3.69%/+9.74%. M1/T4 is an explicit −2.43% wall/−4.32% cycles residual despite 8.77% fewer instructions. All 64 Qwen tokens equal the x86 golden in all nine runs; median assembly vs SDOT TTFT −1.55%, TPOT −0.05% neutral, E2E −0.13%. Disassembly proves GCC's framed dependent one-block loop versus the stack-free independent two-block schedule. Same-file Pi llama.cpp, peak memory and concurrency remain `PENDING`; no competitor-floor binding is claimed. | | 2026-08-06 (`KERNEL-CPU-A76-Q8-DOT` R6 competitor checkpoint; physical RPi5 Cortex-A76; lifecycle remains `GATING`; closing commit: this checkpoint) | Measures the separate four-core A76 same-file llama.cpp floor after the assembly leaf became default. No production code changes. The vllm.cpp nominal p16 request measures 17 input tokens, so the binding competitor uses pp17/tg64/pp17+tg64. A same-text CLI arm verifies 64-token greedy output equality. | Official llama.cpp tag b9892 `ee445f93d` reconstructed under QEMU because historical recorded fork object `237ad9b96` is unavailable; exact recorded anchors match (`quants.c:400`, `arch/arm/quants.c:1076`, `repack.cpp:2725`, `qwen35.cpp`). Local evidence: [Pi competitor record](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md). | **CORRECTNESS PASS, PERFORMANCE NOT MET, `benchmark_binding=true`.** Three clean unthrottled vllm.cpp reps: prefill 12.81 tok/s, decode 2.55 tok/s, output-equivalent E2E 2.46 tok/s, E2E 26,018.39 ms. llama.cpp three-sample p17/tg64/combined: 27.77 / 3.91 / 3.77 tok/s, E2E 16,998.49 ms. vllm.cpp ratios 0.461x prefill / 0.653x decode+E2E; peak RSS wins 2.841 vs 3.747 GiB (24.2% less). Same-text normalized output SHA `a5a630d7...` equal; all vllm performance tokens retain golden SHA `0ec98e...`. Intrusive 50 ms forked sampler run VOID; accepted timing has no sampler, RSS sampled separately at 1 Hz. Next lever: fresh both-engine profile, then BF16 GEMM; M1/T4 and concurrency remain. | | 2026-08-07 (`SERVE-CLI-BENCH` + `KERNEL-SSM-MAMBA`; clean sm_120 exact-chunk transplant; local-4B binding only) | Makes the benchmark use production `AsyncLLM`, then ports vLLM's exact `(sequence, BLOCK_M=8 token chunk)` descriptors into shared GDN step metadata and maps one CUDA register-kernel program to each descriptor. `VT_CONV_EXACT_CHUNKS` defaults ON with a same-binary `=0` rollback; `VT_CONV_REG=0` retains tiled/scalar. | Pinned vLLM engine-core queued dispatch `vllm/v1/engine/core.py:200-231,622-669` and causal-conv descriptor mapping `causal_conv1d.py:15-28,71-79,123-124`; local anchors `examples/bench/bench_core.h`, `gdn_attn.{h,cpp}`, `qwen3_5.cpp`, `ops.{h,cpp}`, `cuda_gdn.cu`; [spike/result](specs/sm120-qwen35-conv-chunking-2026-08-07.md). | **ACCEPTED + REBASED-MAIN REPRODUCED.** Contained rebuild; CPU 6/6, CUDA GDN 66/66·4300, cached 4B 3/3·1672; exact/rollback token files identical. On `3d2581551` over `upstream/main` `48a54141f`, same-binary `nsys` reproduces conv **718.704→233.955 ms = 3.072x** and profiled total **6589.65→6739.34 tok/s = +2.272%**; vLLM 145.421 ms leaves **1.609x**. Binding three-pair A/B remains total/output **+2.152%**, TTFT **-2.945%**, TPOT **-1.920%**, E2E **-2.118%**, VRAM unchanged. Sealed-vLLM throughput **1.021246x PASS**; latency/VRAM OPEN. No gate-model extrapolation. [Evidence](../docs/bench-evidence/qwen35-4b-sm120-main-20260807.md). | +| 2026-08-08 (`row/ENG-RELEASE-BINARIES`; `ENG-RELEASE-BINARIES` W5; draft PR #141; build-time tooling only) | **Versioned deterministic binary-release manifest contract.** Adds schema v1 plus a stdlib-only generator/validator that records the exact host/backend/static boundary, dependencies, clean source/toolchain/CMake flags/test commands, adaptive CPU tiers and probes, ten-SM CUDA fat inventory with independent runtime/AOT evidence, and preview/stable publication proof. Synthetic CPU/CUDA fixtures are canonical contract inputs, never artifact evidence. | Mirrors the metadata/evidence separation of vLLM's pinned release lanes (`.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`) while expressing vllm.cpp's KISS adaptive-CPU/fat-CUDA matrix and explicit external-runtime boundary. No vLLM runtime behavior changes and no release artifact parity is claimed. | RED-first mutation suite 13/13 covers missing/unknown/inconsistent fields, all evidence states, every current x86 tier, all ten SMs/six AOT trees, channel/static/dependency boundaries, duplicate JSON and nondeterminism; accepted release-contract suite remains 30/30. No GPU, archive, runtime, correctness or performance gate ran. | +| 2026-08-08 (`row/ENG-RELEASE-BINARIES`; `ENG-RELEASE-BINARIES` W5 scoped re-review repair; draft PR #141; tests/records only) | **Closes three surviving W5 manifest-contract mutations without changing production behavior.** Direct assertions now keep JSON booleans distinct from both the integer schema type and integer constants, while an independent CUDA manifest mutation removes the named `nvidia-driver` dependency and requires the exact missing-driver diagnostic. The existing `schema_version: true` integration rejection and arbitrary-driver rejection remain. | No upstream runtime analogue or behavior delta: this strengthens the local release metadata gate around the already-implemented JSON-type identity and external NVIDIA driver boundary. | Each production removal is RED for its own reason: deleting the integer/bool exclusion fails the direct type assertion; deleting JSON type identity fails the direct constant assertion; deleting `require_external("nvidia-driver", "driver")` leaves the missing-driver manifest error-free and fails the exact diagnostic assertion. Restored source is 19/19; cumulative fresh-review production removals killed are 10/10; accepted release-contract suite remains 30/30. Fixtures remain byte-identical synthetic inputs. No artifact, archive, runtime, correctness, performance, GPU, download, or release-channel claim advances. | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index b32c4f0b..0de1f20c 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -80,7 +80,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 14 | `ROAD-V1-D5` | LoRA, local KV/weight offload, expert streaming, wider model zoo | [engine matrix](engine-matrix.md), [model matrix](model-matrix.md) | corrected expert-streaming spike accepted (`ENG-EXPERT-STREAM` READY): bank-only safetensors→Marlin bank, fixed contiguous cache slots matching Marlin dense strides, logical→slot remap after explicit router D2H, chunked C` working everywhere `--dit ` did; gated CPU-only (72/72, 54497 post-rebase) on index/name mapping and on the REAL 535-tensor geometry read from a sparse 13-shard release. The DEVICE streamer landed 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`, spec §8.14): one tensor at a time, zero host buffer for the bulk, bit-exact vs the non-streamed reference (73/73, 55203). **ENCODER + THE NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`, spec §8.15)**: the 14-shard bf16 text encoder streams too and `--encoder-only` runs it alone; measured over 233 tokens, Q4_K_M-vs-bf16 conditioning is cos 0.99745 mean / 6.85% rel RMS excl. sink / 3.5 deg median rotation — as much as a one-word prompt edit, but DIFFUSE. Whether the RENDER changes is NOT established (75/75, 55609). This unblocks the bf16-vs-quantized quality A/B; no render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the NVFP4 DiT projections now keep FP4 PACKED and route through the shared `dense_nvfp4::MatmulNvfp4W4A16D` (Marlin W4A16 — vLLM's own forced-a16 selection; SAME kernel as Laguna/dense-Qwen3 NVFP4; no new quant code); fp4-vs-bf16 WIRING gate GREEN (62/62·30039, W4A16 dispatcher runs all 11 quantized GEMMs). **GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** Marlin W4A16 RAN on sm_121a (`dense_gemms==11` default / `marlin_gemms==11` VT_MARLIN_DENSE=0, `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT; fp4 is a MEMORY win (~16 vs ~66 GB), ~0.79–0.83× the bf16 arm per diffusion forward (compute-bound large M; 3.47× faster at small decode-like M). Real-checkpoint fp4-resident t2va e2e RUNS end-to-end (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-768×1344-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render-coherence bug (device VAE decode / denoise), separate from the fp4 speed work. vLLM-Omni serves NO quantized H3 (BF16-only) -> HW/loader-forced-indirect (4×B300 209f render 86.964 s vs 1×GB10 209 s/forward). **2026-08-08 ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** the public 0/1 selector is mapped once to generic `DeviceType`; DSR returns 34→32 with the baseline/allowlist unchanged; CPU compile/fold test pending in CI due shared-disk pressure. | | 15 | `ROAD-V1-D6` | **llama.cpp device breadth folded into scope (user-directed 2026-08-05):** the 11 ggml backends vLLM has no platform for — cann, musa, opencl, openvino, rpc, webgpu, zdnn, zendnn, hexagon, blas, virtgpu — inventoried as `BACKEND-GGML-*`. **SPIKES FIRST:** no implementation before each row's `.agents/specs/.md` clears the spike contract, per the standing directive. vLLM stays the mirror source; llama.cpp is the breadth reference. | [backend matrix](backend-matrix.md) | ☐ per-row spike required | `INVENTORIED` | first spike accepted | -| REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md) | `SPIKE` | Fresh review of PR #129, then W1 cross-family per-source gencode; multi-SM AOT and CPU ISA audits precede bundle work; no archive exists | +| REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md) | `ACTIVE` | W5 versioned manifest schema/generator and fail-closed CPU/CUDA/backend/channel fixtures implemented in draft PR #141 (19/19; ten fresh-review production removals killed); W1-W4 and W6-W13 remain pending, including unavailable x86 CUDA-13 ten-SM fat-build hardware/toolchain, install/archive/publish work and every real archive/runtime/correctness/performance claim; no archive exists | | IMG | `ROAD-V1-CONTAINERS` | **Published container images on GHCR, built by GitHub Actions (user-directed 2026-08-08).** The same staged bundle `ROAD-V1-RELEASE` defines, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane — and `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image, every supported SM), `vulkan`, `cpu`, plus `rocm` blocked-preview; version tags immutable, every `latest-` moves. Every lane is a `linux/amd64`+`linux/arm64` multi-arch manifest on native runners, because the project's own gate hardware (GB10, Thor, Orin) is arm64. Metal and MLX are NOT-CONTAINERIZABLE and stay static-binary-only — a recorded boundary, not pending work. Depends on the `ROAD-V1-RELEASE` install/stage tree: the image IS the bundle, so the two lanes must not grow separate layouts. No image, workflow or registry package exists. | [`ENG-RELEASE-CONTAINERS`](engine-matrix.md) | none yet: `planned: specs/container-images.md`; the row cannot leave `INVENTORIED` without it | `INVENTORIED` | Write the container-image spike spec (lane matrix, base images, manifest/provenance, retention, least-privilege publish flow) against the accepted release-binary contract; no image work starts before it | An area row cannot enter `READY` without a real spike under `specs/`, and cannot diff --git a/.agents/specs/release-binary-matrix.md b/.agents/specs/release-binary-matrix.md index a0ec7caa..2f7d548c 100644 --- a/.agents/specs/release-binary-matrix.md +++ b/.agents/specs/release-binary-matrix.md @@ -1,17 +1,20 @@ # Downloadable server binary release matrix -Status: accepted spike for `ENG-RELEASE-BINARIES`. This document defines the -release contract and helper-sized implementation order. It does not claim that -an archive, release workflow, or runtime result exists. +Status: accepted contract with W5 manifest tooling implemented for +`ENG-RELEASE-BINARIES`. The release row is `ACTIVE`; no archive, release +workflow, or runtime result is claimed. Pins: vLLM parity source `555967922`; vllm.cpp spike baseline `f13c49ee`; request [#117](https://github.com/mudler/vllm.cpp/issues/117); claim `CLAIM-ENG-RELEASE-BINARIES-SPIKE` in draft PR -[#129](https://github.com/mudler/vllm.cpp/pull/129). +[#129](https://github.com/mudler/vllm.cpp/pull/129); W5 implementation claim +`CLAIM-ENG-RELEASE-BINARIES-W5` in draft PR +[#141](https://github.com/mudler/vllm.cpp/pull/141). identity=ENG-RELEASE-BINARIES -lifecycle=SPIKE +lifecycle=ACTIVE +manifest_schema=vllm.cpp.release-manifest.v1 primary_cuda_artifact=one-fat-binary-per-os-host-abi primary_cuda_sms=80,86,87,89,90a,100a,103a,110,120a,121a per_sm_cuda=optional-non-primary @@ -27,12 +30,13 @@ musl_channel=experimental-preview musl_scope=cpu-only-no-gpu rocm_channel=blocked gpu_driver_boundary=external-host-never-bundled -required_anchor_paths=.agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md,.agents/coordination.md,.agents/state.md,docs/STATUS.md,docs/BENCHMARKS.md +required_anchor_paths=.agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md,.agents/coordination.md,.agents/state.md,docs/STATUS.md,docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json,scripts/release_manifest.py,tests/scripts/test_release_manifest.py work_W1= work_W2=W1 work_W3= work_W4= work_W5= +work_W5_status=implemented work_W6= work_W7=W1,W2,W3,W4,W5,W6 work_W8=W5,W7 @@ -98,11 +102,15 @@ installed server archives while vLLM remains the runtime-behavior oracle. ## Our baseline The current server is the CMake target `server`, linked to the static `vllm` -target and gated only by a help smoke (`examples/CMakeLists.txt:54-64`). The +target and gated only by a help smoke (`examples/CMakeLists.txt:54-64`). W5 now +provides the versioned schema and deterministic build-time generator/validator +(`release/manifest-v1.schema.json`, `scripts/release_manifest.py`) with +synthetic CPU/CUDA fixtures; those fixtures are contract tests, not artifact or +runtime evidence. The library install rules package `libvllm.a`, the shared C ABI library, and `include/vllm.h` (`CMakeLists.txt:1712-1783`). There is no server install rule, -archive layout, release manifest, staged dependency audit, provenance, or -publish workflow. Existing CUDA AOT trees are exactly the six directories named +archive layout, staged dependency audit, provenance, or publish workflow. +Existing CUDA AOT trees are exactly the six directories named below. Cross-family CUDA fat builds currently fail because feature sources are gencode'd for the whole list, and `cmake/TritonAOT.cmake:94-126` currently rejects multi-arch AOT; both are prerequisites, not accepted release limits. @@ -423,8 +431,8 @@ wildcards, and a failed lane cannot be replaced by an older workflow artifact. ## Work breakdown: helper-sized implementation plan Each work unit is a separate claim with its own red-first checker change and -fresh review. No unit advances this row beyond `ACTIVE` until its gates pass; -the present checkpoint stays `SPIKE`. +fresh review. W5 is implemented in draft PR #141 and advances the row to +`ACTIVE`; no later work unit or release evidence is implied. | Work | Deps | Deliverable | Exit gate | |---|---|---|---| @@ -432,7 +440,7 @@ the present checkpoint stays `SPIKE`. | W2 | W1 | multi-SM Triton AOT embedding, namespacing, manifest and exact runtime dispatch | all six available trees coexist in one fat binary; exact-SM dispatch tests plus portable fallback for the four unavailable trees; wrong-tree mutation red | | W3 | — | x86_64 CPU ISA-dispatch inventory and completion | SSE2/portable baseline runs without AVX2; current F16C/AVX2/AVX-512 tiers forced and executed; exact OS-state probes; VNNI/AMX listed only for real gated kernels; no `-march=native` | | W4 | — | aarch64 CPU ISA-dispatch inventory and completion | NEON/portable baseline plus independently forced DotProd/i8mm where kernels exist; exact Linux HWCAP/Darwin sysctl gates; poor/rich host or emulation execution | -| W5 | — | versioned release-manifest generator and schema with independent per-SM and per-CPU-tier evidence | fixtures distinguish absent, false, failed and true; compiled tiers/SMs, required probes, dependencies and host ABI are mandatory | +| W5 | — | **IMPLEMENTED (#141):** versioned release-manifest generator and schema with independent per-SM and per-CPU-tier evidence | 19/19 tests: fixtures distinguish absent, false, failed and true; schema type/enum/const are independently live; booleans cannot satisfy integer types/constants; CPU tier kernel-family/bit/OS-probe inventories are exact; compiled tiers/SMs, fallback/AOT state, the named NVIDIA driver dependency, publication channels, static boundaries and host ABI are mandatory | | W6 | — | canonical `vllm-server` output name, install component, and staging/package target for the existing static-core server | install into empty prefix; extracted help smoke; existing library install unchanged | | W7 | W1, W2, W3, W4, W5, W6 | staged archive validator: allowlist, dependency/RPATH, fat-SM/AOT and adaptive-CPU audits, SHA256, VERSION, licenses and SPDX SBOM | Linux fixture/archive tests red-first; no build paths, missing SM, unsafe ISA tier or undeclared dependency accepted | | W8 | W5, W7 | least-privilege dry-run/tag workflow, immutable artifact handoff, provenance and protected publish stages | permissions checker plus dry run proves no release is created and publish cannot consume unverified bytes | @@ -474,6 +482,7 @@ The release program is feasible as backend-specific static-core bundles with a hybrid stable/preview channel. Literal-static scope is limited to the experimental musl CPU lane. ROCm is blocked. Primary downloads are adaptive CPU or fat CUDA per OS+host ABI; per-SM CUDA archives are optional diagnostics. The -first implementation slice is W1's cross-family per-source-gencode prerequisite, -followed by multi-SM AOT dispatch and CPU ISA audits before any bundle workflow. -`ENG-RELEASE-BINARIES` is `SPIKE`, not `READY`, `ACTIVE`, or `DONE`. +W5 manifest schema/tooling is implemented and the row is `ACTIVE`, not `DONE`. +W1-W4 and W6-W13 remain pending; in particular no archive, install/package or +publish workflow, staged-archive evidence, runtime/correctness proof, or +performance result exists. diff --git a/.agents/state.md b/.agents/state.md index 93a8d0da..d25d7498 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -43120,6 +43120,21 @@ F79-4 remain open on the landed tree; the review's merge-and-fix map is the binding description. Pi concurrency, BF16 GEMM/speed closure (W6) stay open as the lane's own next steps. +## 2026-08-08 — Release manifest W5 claimed after #129 merged + + +PR #129 merged the accepted `ENG-RELEASE-BINARIES` spike at `b38f78a7`. +The issue/PR search found no duplicate open release implementation claim, and +the current tree still has no versioned manifest schema, generator, validator, +or manifest fixtures. `CLAIM-ENG-RELEASE-BINARIES-W5` now owns only that W5 +slice in `/home/mudler/_git/vllm.cpp-release-binaries` on +`row/ENG-RELEASE-BINARIES`; a new draft PR opened from this checkpoint is the +claim. Implementation has not started. The row stays `SPIKE`, and every archive, +runtime, correctness, performance, CUDA-fat, CPU-tier, install, and publication +claim remains pending. Next: add deterministic RED-first fixtures that prove +the four evidence states and reject missing or inconsistent host/backend, +CPU-tier, CUDA-SM, dependency, channel, and static-boundary metadata. + ## 2026-08-08 — ROCm approach-(b): unified memory true by construction on integrated APUs @@ -43242,6 +43257,27 @@ memberships (incl. `MistralModel`) unregistered. (3) `/pooling`, `/score`, (5) `vllm_embed` batches sequentially through the synchronous engine (recorded deviation; AsyncLLM stays generation-only). +## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE + + +Draft PR #141 implements only `ENG-RELEASE-BINARIES` W5: versioned schema +`vllm.cpp.release-manifest.v1`, a deterministic stdlib-only build-time +generator/validator, and canonical synthetic CPU/CUDA fixtures. The 13-test +suite mutates schema identity, required and unknown fields, all four evidence +states, every current x86 CPU tier, all ten primary CUDA SMs and six available +AOT mappings, backend flags, preview/stable publication evidence, dependencies, +static boundaries, host ABI and duplicate/non-deterministic inputs. CI and +preflight register it fail-closed; the accepted release-contract suite remains +green at 30/30. + +This is contract tooling, not release evidence. Fixture `passed` states are +synthetic and cannot satisfy an artifact gate. W1-W4 and W6-W13 remain pending; +there is no server install/package target, archive, staged-archive result, +runtime/correctness/performance result, SBOM/provenance for published bytes, or +publication workflow. The required x86 CUDA-13 ten-SM fat-build hardware and +toolchain are unavailable in this worktree. The row advances from `SPIKE` to +`ACTIVE`, never `DONE` or `GATING`. + ## 2026-08-08 — PR #140 ROCm gfx1201 + Gemma-4-26B-A4B MoE landed with contributor authorship preserved + operator hardening @@ -43313,6 +43349,33 @@ Row moved INVENTORIED→READY (roadmap_v1 + feature-matrix §2). M1-M4 are the i rows this unblocks; M1/M2/M4-CPU are CPU-completable, M3 (profile run) is dgx-gated. Records-only spike (no code). +## 2026-08-08 — Release manifest W5 fresh-review contract repair + + +Fresh mutation review of draft PR #141 found seven independently surviving +production removals: schema type, enum and const enforcement; stable +runtime/correctness evidence; required CPU bits; CUDA portable-fallback +mapping; and CUDA AOT evidence-state rules. It also proved the unmodified +validator accepted `schema_version: true`, and that CPU manifests could invent +kernel families or add CPU bits and OS probes. + +The repair makes schema equality JSON-type-strict (so booleans never equal +integer constants), declares `schema_version` as integer const 1, and binds the +current x86_64 and aarch64 CPU tiers to exact kernel-family, CPU-bit and OS-probe +inventories grounded in the compiled selectors. Linux aarch64 uses +`getauxval(AT_HWCAP2) & HWCAP2_I8MM`; Darwin uses +`sysctl:hw.optional.arm.FEAT_I8MM`. No independent VNNI, AMX, BF16 or DotProd +tier is advertised because no separately gated current kernel tier exists. +The focused suite is 17/17, and isolated mutation runs kill all seven original +production-removal classes. Four readable JSON goldens moved byte-for-byte +under `tests/scripts/fixtures/` so CI's documented fixture exemption applies; +the checker and fixtures were not compacted or weakened. + +This remains W5 contract tooling with synthetic fixtures. W1-W4 and W6-W13, +all archives and staged smoke, real runtime/correctness/performance evidence, +and publication remain pending. `ENG-RELEASE-BINARIES` stays `ACTIVE`; no +artifact, backend, runtime, performance or release-channel claim advances. + ## 2026-08-08 — ROAD-V1-MEM M1+M2 LANDED: absolute --kv-cache-memory knob + group-aware bytes-per-block (CPU brick) @@ -43333,6 +43396,31 @@ MLA no-factor-2 / hybrid-excludes-mamba / het-KV per-layer / divisor), clean -Werror. M3 (the `gpu_memory_utilization` device profile run) stays dgx-gated; until it lands the util branch falls back to 256. Row `M1+M2 DONE`. +## 2026-08-08 — Release manifest W5 scoped re-review test repair + + +Scoped re-review found that two Python/JSON identity guards and the required +CUDA NVIDIA-driver declaration were enforced in production but not each pinned +by an independent test. Two named tests close those holes. One directly asserts +that JSON `true` is neither the integer schema type nor equal to integer constant +`1`; the other removes `nvidia-driver` from an otherwise valid CUDA manifest and +requires the exact missing-driver diagnostic. The existing integration rejection +of `schema_version: true` and arbitrary-driver rejection remain intact. + +All three source-removal mutations are independently red: removing the bool +exclusion fails only the direct integer-type assertion, removing JSON type +identity fails only the direct constant assertion, and removing the named +`require_external` call leaves the missing-driver manifest error-free and fails +the exact diagnostic assertion. Restored production is 19/19; the cumulative +fresh-review removal battery is 10/10, with the accepted release contract still +30/30. + +This is tests-and-records hardening over unchanged W5 tooling. Fixtures remain +byte-identical and synthetic. W1-W4 and W6-W13, all archives and staged smoke, +real runtime/correctness/performance evidence, and publication remain pending. +`ENG-RELEASE-BINARIES` stays `ACTIVE`; no artifact or release-channel claim +advances. + ## 2026-08-08 — TP-W1 LANDED: rank-layout group table (parallel_layout.h) + LoadedModel TP handle @@ -43356,6 +43444,26 @@ additive/byte-neutral). Header-only, no CMake wiring beyond the test. Next: TP-W (row/input-dim shard + rank-0 bias + per-rank Hq/Hkv + QKV kv-replication + vocab embed/lm_head + logits all-gather). +## 2026-08-08 — clean-CI live-row audit and embedding lifecycle repair + + +PR #141 exposed a host-dependent record gate: `audit-live-rows.py` counted +local `refs/heads/row/*` as repo-wide claim evidence, so this workstation's +stale local `row/MODEL-EMBED-llama-llama-for-causal-lm` ref made preflight +green while a clean GitHub checkout correctly reported that ACTIVE row +ABANDONED. A RED-first unit test now proves local-only refs are ignored; only +fetched remote `row/*` refs can materialize shared in-flight evidence. + +With the false local evidence removed, the record's real lifecycle error was +visible. The ROW 6 implementation is already on main at `57ed063e`, but only +`LlamaModel` of eight memberships is registered and the real-checkpoint +`vllm.LLM(task="embed")` cosine gate has not run. Per the lifecycle contract, +the model row is therefore `PARTIAL`, not `ACTIVE`; its checklist counts, +closed-PR wording, coordination scope, NOW snapshot and all three obligated +public surfaces now say so. No implementation, runtime result or release W5 +artifact claim changes. The focused audit suite is 42/42 and the clean audit +reports 0 abandoned ACTIVE rows. + ## 2026-08-08 — FIX: device-leakage regression on main (async readback becomes a Backend capability) — unblocks 3 contributor PRs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ea669451..04a458ea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -48,6 +48,9 @@ jobs: run: | python3 scripts/check-release-binary-contract.py python3 tests/scripts/test_check_release_binary_contract.py + - name: Binary release manifests are deterministic and fail closed + run: | + python3 tests/scripts/test_release_manifest.py - name: README stays a human-readable user-facing document run: | python3 scripts/check-readme-structure.py diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 650a485f..8578ca0e 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,10 +1,10 @@ # Benchmarks -## At a glance +## At a glance: W5 manifest guarded; release artifacts pending | Reference | Workload | Headline | Tokens | |---|---|---|---| -| **Binary release matrix (spiked)** | `ENG-RELEASE-BINARIES`: primary host-ABI fat-CUDA + adaptive-CPU static-core bundles; optional per-SM diagnostics; experimental literal-static musl CPU | **PENDING:** pins 10-SM fat CUDA, adaptive no-AVX2 CPU, W1-W13/W10-W12 policy, public pending states; 30 tests GREEN. No archive, staged smoke, runtime, correctness, or performance evidence | n/a | +| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`: versioned deterministic manifest for primary host-ABI fat-CUDA + adaptive-CPU static-core bundles, optional per-SM diagnostics and experimental literal-static musl CPU | **W5 GREEN:** 19/19 + ten review mutations killed; contract 30/30; PR-size classes/budget GREEN. Synthetic fixtures only. **PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates | n/a | | **Container images (inventoried)** | `ENG-RELEASE-CONTAINERS`: GHCR images from GitHub Actions; lanes `-cuda`/`-vulkan`/`-cpu`, amd64+arm64 manifests | **No number owed:** record-only. No Dockerfile, workflow, registry package or image exists; the image is the unimplemented `ENG-RELEASE-BINARIES` bundle | n/a | | **Developer agent entry point (implemented)** | `DOCS-AGENT-PROTOCOL-ENTRYPOINT`: public contribution guide + synchronized, mutation-gated pre-claim intake rule | Rebased documentation/protocol only; benchmark void | n/a | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | @@ -383,12 +383,3 @@ built on it rather than keeping the flattering one. Build flags, environment variables, and the full gate list are in [BUILD.md](BUILD.md) and [ENVIRONMENT.md](ENVIRONMENT.md). - -## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700) - -| Path | Warm tok/s | Notes | -|------|------------|--------| -| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill | -| server `/v1/completions` | ~38 | exclusive | -| server `/v1/chat` thinking off | ~32 | after expert cache | -| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 5a779868..c7c97a29 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -27,6 +27,8 @@ These change how the engine runs and have no CLI flag (or complement one). | `VT_VULKAN_DEVICE` | first suitable device | Forces the Vulkan physical device index. Required on a multi-GPU host to pin the intended device | | `VT_KV_CACHE_F32` | off (native KV dtype) | Forces the KV cache to fp32. A precision/diagnostic lever, at the cost of double the KV memory | | `VT_ENABLE_JUMP_FORWARD` | off | Opt-in to jump-forward constrained decoding (SGLang parity SW3): when a grammar/structured-output request reaches a state with exactly one valid next token, that token is emitted without a model step. Currently drives only the standalone driver (`DrainForcedTokens`); output-identical by construction (it fires only where the constrained sampler already has a single valid token), so it changes speed, never tokens. Off by default until the production scheduler splice (jumped-token KV recompute) lands. Set `1`/`true`/`on` to enable | +| `VT_SERVER_MAX_PROMPT_CHARS` | `200000` | Rejects larger `/v1/chat/completions` prompts before scheduling. `0` disables the guard. This is a character count after chat-template rendering, not a token limit | +| `VT_SERVER_MAX_NEW_TOKENS` | `4096` | Caps the request's `max_tokens` value for `/v1/chat/completions`. `0` disables the cap | ## GGUF loading @@ -118,6 +120,7 @@ Read-only observability; none change output. | `VT_DFLASH_GRAPH_STATS` | unset | Print DFlash draft-step CUDA-graph capture/replay counts to stderr | | `VT_OP_PROVIDER_STATS` | off | Print per-op provider (which backend served each op) statistics | | `VT_OP_PROVIDER_DISABLE` | (none) | Comma-separated provider names to disable, forcing fallback (diagnostic) | +| `VT_SERVER_PREFILL_PROGRESS` | off | `=1` prints chunked-prefill progress to stderr, rate-limited to roughly 2 Hz per request. `=0` explicitly disables it even when `VT_SERVER_VERBOSE=1` | | `VT_GDN_VALIDATE` | off | Run the GDN validation/cross-check path (slower; for kernel debugging) | | `VT_FP4_AUTOTUNE_VERBOSE` | off | Log the NVFP4 GEMM autotuner's tactic selection | | `VT_H3_PROGRESS` | unset | Trace the MiniMax-H3 denoise loop's phases to stderr: which forward path was taken (device vs the CPU reference), how long the ONE-TIME device weight staging took, and per-step forward seconds with the sequence length. A real-checkpoint run spends its minutes in exactly one of those phases, and this says which without guessing — it was added after GPU-utilization counters proved unreliable on Tegra-class boards | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 4915d9fb..22c75237 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -26,6 +26,7 @@ are our reading of their documented behavior, not measurements. | Weight formats | Safetensors + GGUF | Safetensors | Safetensors | GGUF | | Correctness gate | token-exact vs vLLM | reference | own | own | | Architectures | 30 registered, 25+ gated | 130+ | 100+ | 100+ | +| Downloadable server binaries | ◐ W5 mutation-gated manifest contract implemented; no archives published | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries | ## Serving and scheduling diff --git a/docs/STATUS.md b/docs/STATUS.md index bb6a6964..27fe69cd 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -122,7 +122,7 @@ token-for-token correctness against the pinned oracle. | Tool-call parsing | 36 parser families / 40 accepted names, streaming | Every vLLM tool parser at the pin except the three Rust/Harmony-backed ones: pure-text parsers ported 1:1, the six engine-backed families reimplemented from their wire formats, all held to the upstream test suites. Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md | | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | | Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md | -| OpenAI server | Supported (subset); #129: SPIKE∅ | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`, conditioned on an `input_reference` image or the `metadata` video/audio references. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | +| OpenAI server | Subset; #141 W5 19/19 ACTIVE ART∅ SIZE✓ | Completion/chat (streaming SSE), models, health/version/ping, metrics, tokenize/detokenize, tokenizer/server info, prefix-cache reset, abort, and Sora-shaped video creation/content. Tokenizer info and abort are flag-gated; metrics and cache reset lack live async backing. Details: docs/USAGE.md | | Pooling task class (embeddings / classify / score / rerank) | **EMBEDDINGS LIVE ON THE ONE SURFACE (ROW 6)**: `LlamaModel` registered, `PoolingRunner` in the engine step, `vllm_embed` (ABI v15) + live `/v1/embeddings`; classify/score/rerank engine-side only | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** (`EmbeddingPoolerHead`, `ClassifierPoolerHead`, `SequencePooler` + factories, `DispatchPooler` routing, `PoolerConfig`/`PoolingParams`; `test_pooler_heads` 27/27-240, RED-first). **W3 landed the pooling RUNNER path** (`PoolingRunner`: pooled embeddings instead of sampled tokens, structural cosine gate vs an f64 LAST+normalize reference, `test_pooling_runner` 5/5-14, RED-first). **ROW 6 (2026-08-08): embeddings LIVE** — fold gate `test_llama_embedding_fold` 4/4-231 (engine path == direct registry path, f64 LAST+normalize ref, chunked is_valid arm); residuals: REAL checkpoint + `LLM(task="embed")` oracle cosine (no number fabricated), score/rerank/classify endpoints, matryoshka/base64/token-array inputs, tokwise (W5). Detail: `.agents/specs/embeddings-one-surface.md` | | Plugin system (out-of-core registration) | Spiked; first CPU brick landed, not yet wired into any production path | The extensibility-first discovery layer. W0 spike over vLLM's plugin surface (general / platform / io_processor / endpoint groups, the `register_model` an out-of-tree plugin calls, the invocation seams) is committed (`.agents/specs/plugin-system.md`, `ENG-PLUGIN-SYSTEM` ACTIVE, `CLAIM-PLUGIN-SYSTEM`). W1 landed `vllm::plugins::LoadGeneralPlugins()` + the out-of-core general-plugin registration seam (`RegisterGeneralPlugin` / `REGISTER_VLLM_GENERAL_PLUGIN`) over the existing `REGISTER_VLLM_MODEL`-style registries (the in-tree factory `MODEL-FACTORY-registry` is record-repaired `DONE` 2026-08-05: 28 self-registering TUs, dgx debt paid by the 2026-07-23 seven-gate run): a 1:1 mirror of `load_general_plugins` (load-once idempotence, the `VLLM_PLUGINS` allowlist, per-plugin failure isolation). Proven by an out-of-core toy-model plugin that registers a toy architecture through the public `RegisterModel` seam — unit-gated RED-first (`test_plugin_system` 1 case / 29 assertions: the toy arch resolves ONLY after LoadGeneralPlugins runs it, and not under `VLLM_PLUGINS=""`). Python entry points have no C++20 analogue, so discovery is the project's static-init/`dlopen` registration idiom (recorded porting-inventory §9). NOT yet wired: real shared-object `dlopen` + the C-ABI `vllm_plugin_register` entry (W2), the engine/CLI `--load-plugins` wiring that calls LoadGeneralPlugins from the construction paths (W3), the platform/quant plugin kinds (W4), and the io_processor/stat_logger/endpoint groups (W5) are named residuals. See docs/BENCHMARKS.md | | Offline Batch API (JSONL file runner) | Spiked; first CPU brick landed, not yet exposed as a CLI | The offline OpenAI Batch API: read a JSONL of OpenAI-format requests, run each through the engine, write a JSONL of responses. W0 spike over vLLM's `run_batch.py` (schema, endpoint dispatch, run loop, file I/O) is committed (`.agents/specs/batch-api.md`, `SERVE-BATCH-API` ACTIVE, `CLAIM-BATCH-API`). W1 landed `RunBatch` (`RunLine`/`RunLines`/`Run`) + `RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map: `/v1/chat/completions` dispatch, the `BatchResponseData`/`BatchRequestOutput` schema (`vllm-` ids, custom_id echoed), the `run_request` AllResponse/ErrorResponse/stream branches, and the unsupported-endpoint/url error rows. Unit-gated RED-first (`test_openai_run_batch` 7 cases / 80 assertions over the synthetic serving engine: ordered rows + custom_id echo + per-line BatchRequestOutput schema round-trip, a malformed line isolated into an error row so the batch continues, dispatch + 404 error rows; dropping the custom_id echo fails 9 assertions). Recorded deviation: a malformed line is isolated (batch continues) where upstream aborts the job. NOT yet exposed: the `vllm run-batch` CLI + `BatchFrontendArgs` (W2), embeddings/score/rerank dispatch (W3, rides pooling endpoints), audio transcription/translation + media fetch (W4), and http(s)/data-URL file I/O + metrics + overlapped `AsyncLLM` submission (W5) are named residuals. See docs/BENCHMARKS.md | @@ -1319,7 +1319,7 @@ LocalAI house style (side-by-side, identical output, honest measured ratios). ## Backend detail -Gemma4 ROCm fused helpers route through the portable `include/vt/fused_ops.h` seam rather than calling `vt::rocm::*` from model files; ROCm fast path under `VLLM_CPP_HIP`, non-HIP stubs for peer/pin/resident upload, `check-device-leakage` at baseline (#154). +Gemma4 ROCm fused helpers use portable `include/vt/fused_ops.h`: ROCm fast paths under `VLLM_CPP_HIP`; non-HIP stubs cover peer/pin/resident upload; kernel-only `VT_GEMMA4_*` switches are classified and device leakage stays at baseline (#154). **Platform SELECTION is the one non-additive site, and is now gated.** A platform missing from `CurrentPlatform()`'s hardcoded walk registers and answers diff --git a/release/manifest-v1.schema.json b/release/manifest-v1.schema.json new file mode 100644 index 00000000..8450bfa3 --- /dev/null +++ b/release/manifest-v1.schema.json @@ -0,0 +1,317 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/mudler/vllm.cpp/release/manifest-v1.schema.json", + "title": "vllm.cpp binary release manifest v1", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "schema_version", + "artifact", + "host", + "backend", + "dependencies", + "build", + "supply_chain", + "evidence" + ], + "properties": { + "schema": { + "const": "vllm.cpp.release-manifest.v1" + }, + "schema_version": { + "type": "integer", + "const": 1 + }, + "artifact": { + "$ref": "#/$defs/artifact" + }, + "host": { + "$ref": "#/$defs/host" + }, + "backend": { + "$ref": "#/$defs/backend" + }, + "dependencies": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/dependency" + } + }, + "build": { + "$ref": "#/$defs/build" + }, + "supply_chain": { + "$ref": "#/$defs/supplyChain" + }, + "cpu": { + "$ref": "#/$defs/cpu" + }, + "cuda": { + "$ref": "#/$defs/cuda" + }, + "evidence": { + "$ref": "#/$defs/evidenceSet" + } + }, + "$defs": { + "artifact": { + "type": "object", + "additionalProperties": false, + "required": ["id", "version", "channel", "kind", "static_boundary"], + "properties": { + "id": {"type": "string", "minLength": 1}, + "version": {"type": "string", "minLength": 1}, + "channel": { + "enum": ["stable", "preview", "experimental-preview"] + }, + "kind": {"enum": ["primary", "diagnostic"]}, + "static_boundary": {"enum": ["static-core", "literal-static"]} + } + }, + "host": { + "type": "object", + "additionalProperties": false, + "required": ["os", "arch", "abi", "abi_version"], + "properties": { + "os": {"enum": ["linux", "macos"]}, + "arch": {"enum": ["x86_64", "aarch64"]}, + "abi": {"enum": ["glibc", "musl", "macos"]}, + "abi_version": {"type": "string", "minLength": 1} + } + }, + "backendFlags": { + "type": "object", + "additionalProperties": false, + "required": [ + "MLX_ROOT", + "VLLM_CPP_BUILD_EXAMPLES", + "VLLM_CPP_BUILD_TESTS", + "VLLM_CPP_CUDA", + "VLLM_CPP_CUDA_ARCHITECTURES", + "VLLM_CPP_HIP", + "VLLM_CPP_HIP_ARCHITECTURES", + "VLLM_CPP_METAL", + "VLLM_CPP_MLX", + "VLLM_CPP_SERVER", + "VLLM_CPP_TRITON", + "VLLM_CPP_VULKAN" + ], + "properties": { + "MLX_ROOT": {"type": "string"}, + "VLLM_CPP_BUILD_EXAMPLES": {"type": "boolean"}, + "VLLM_CPP_BUILD_TESTS": {"type": "boolean"}, + "VLLM_CPP_CUDA": {"type": "boolean"}, + "VLLM_CPP_CUDA_ARCHITECTURES": { + "type": "array", + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "VLLM_CPP_HIP": {"type": "boolean"}, + "VLLM_CPP_HIP_ARCHITECTURES": { + "type": "array", + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "VLLM_CPP_METAL": {"type": "boolean"}, + "VLLM_CPP_MLX": {"type": "boolean"}, + "VLLM_CPP_SERVER": {"type": "boolean"}, + "VLLM_CPP_TRITON": {"type": "boolean"}, + "VLLM_CPP_VULKAN": {"type": "boolean"} + } + }, + "resolvedFeature": { + "type": "object", + "additionalProperties": false, + "required": ["name", "compiled_sms"], + "properties": { + "name": {"type": "string", "minLength": 1}, + "compiled_sms": { + "type": "array", + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + } + } + }, + "backend": { + "type": "object", + "additionalProperties": false, + "required": ["name", "flags", "resolved_features", "gpu_driver_boundary"], + "properties": { + "name": {"enum": ["cpu", "cuda", "metal", "mlx", "vulkan"]}, + "flags": {"$ref": "#/$defs/backendFlags"}, + "resolved_features": { + "type": "array", + "items": {"$ref": "#/$defs/resolvedFeature"} + }, + "gpu_driver_boundary": { + "enum": ["not-applicable", "external-host-never-bundled"] + } + } + }, + "dependency": { + "type": "object", + "additionalProperties": false, + "required": ["name", "version", "kind", "linkage", "bundled", "role"], + "properties": { + "name": {"type": "string", "minLength": 1}, + "version": {"type": "string", "minLength": 1}, + "kind": {"enum": ["library", "framework", "driver", "tool"]}, + "linkage": {"enum": ["static", "dynamic", "external"]}, + "bundled": {"type": "boolean"}, + "role": {"enum": ["build-time", "runtime", "external-runtime"]} + } + }, + "build": { + "type": "object", + "additionalProperties": false, + "required": [ + "source_commit", + "source_clean", + "compiler", + "toolchain", + "resolved_cmake_options", + "test_commands" + ], + "properties": { + "source_commit": {"type": "string", "minLength": 40}, + "source_clean": {"type": "boolean"}, + "compiler": {"type": "string", "minLength": 1}, + "toolchain": {"type": "string", "minLength": 1}, + "resolved_cmake_options": {"$ref": "#/$defs/backendFlags"}, + "test_commands": { + "type": "array", + "minItems": 1, + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + } + } + }, + "evidence": { + "type": "object", + "additionalProperties": false, + "required": ["state", "reason", "command", "result", "url"], + "properties": { + "state": {"enum": ["absent", "not-applicable", "failed", "passed"]}, + "reason": {"type": "string"}, + "command": {"type": "string"}, + "result": {"type": "string"}, + "url": {"type": "string"} + } + }, + "evidenceSet": { + "type": "object", + "additionalProperties": false, + "required": [ + "build", + "archive_smoke", + "dependency_audit", + "runtime", + "correctness", + "performance" + ], + "properties": { + "build": {"$ref": "#/$defs/evidence"}, + "archive_smoke": {"$ref": "#/$defs/evidence"}, + "dependency_audit": {"$ref": "#/$defs/evidence"}, + "runtime": {"$ref": "#/$defs/evidence"}, + "correctness": {"$ref": "#/$defs/evidence"}, + "performance": {"$ref": "#/$defs/evidence"} + } + }, + "supplyChain": { + "type": "object", + "additionalProperties": false, + "required": ["archive_checksum", "sbom", "provenance", "licenses"], + "properties": { + "archive_checksum": {"$ref": "#/$defs/evidence"}, + "sbom": {"$ref": "#/$defs/evidence"}, + "provenance": {"$ref": "#/$defs/evidence"}, + "licenses": {"$ref": "#/$defs/evidence"} + } + }, + "cpuTier": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "kernel_families", + "required_cpu_bits", + "required_os_state", + "execution_evidence" + ], + "properties": { + "name": {"type": "string", "minLength": 1}, + "kernel_families": { + "type": "array", + "minItems": 1, + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "required_cpu_bits": { + "type": "array", + "minItems": 1, + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "required_os_state": { + "type": "array", + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "execution_evidence": {"$ref": "#/$defs/evidence"} + } + }, + "cpu": { + "type": "object", + "additionalProperties": false, + "required": ["baseline", "selected_tier", "compiled_tiers"], + "properties": { + "baseline": {"type": "string", "minLength": 1}, + "selected_tier": {"type": "string", "minLength": 1}, + "compiled_tiers": { + "type": "array", + "minItems": 1, + "items": {"$ref": "#/$defs/cpuTier"} + } + } + }, + "smEvidence": { + "type": "object", + "additionalProperties": false, + "required": [ + "sm", + "aot_available", + "portable_fallback", + "aot_evidence", + "runtime_evidence" + ], + "properties": { + "sm": {"type": "string", "minLength": 1}, + "aot_available": {"type": "boolean"}, + "portable_fallback": {"type": "boolean"}, + "aot_evidence": {"$ref": "#/$defs/evidence"}, + "runtime_evidence": {"$ref": "#/$defs/evidence"} + } + }, + "cuda": { + "type": "object", + "additionalProperties": false, + "required": ["compiled_sms", "sm_evidence"], + "properties": { + "compiled_sms": { + "type": "array", + "minItems": 1, + "items": {"type": "string", "minLength": 1}, + "uniqueItems": true + }, + "sm_evidence": { + "type": "array", + "minItems": 1, + "items": {"$ref": "#/$defs/smEvidence"} + } + } + } + } +} diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 297994d6..3b06b0d3 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -81,6 +81,7 @@ SUITES=( test_agent_gates test_agent_record test_check_release_binary_contract + test_release_manifest test_agent_role test_agent_onboard test_agent_start diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 68f6bd2e..d6dff139 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -317,6 +317,19 @@ # Bumped for a real new row, never to make a failing state transition pass. ENGINE_ROWS = 144 +ENGINE_SUMMARY_SECTIONS = ( + ("Engine and scheduling", "Engine core and scheduling"), + ("KV cache and memory", "KV cache and memory"), + ("Parallelism", "Parallelism and scale-out"), + ("Sampling and generation", "Sampling and generation controls"), + ("Structured output and tools", "Structured outputs and tool calling"), + ("Speculative decoding", "Speculative decoding"), + ("Serving, API, CLI, library", "Serving surface, CLI, and library"), + ("LoRA and adapters", "LoRA and adapters"), + ("Long context and attention", "Long context and attention breadth"), + ("Loading, tokenizer, config", "Loading, tokenizer, and config"), +) + MATRIX_PATHS = [ENGINE_MATRIX, *(path for path, _ in MATRICES.values())] REQUIRED = [ ROOT / "AGENTS.md", @@ -591,33 +604,66 @@ def check_engine_summary(rows: list[ClaimRow], errors: list[str]) -> None: lines = ENGINE_MATRIX.read_text(encoding="utf-8").splitlines() header: list[str] | None = None total: list[str] | None = None - for line in lines: + summaries: dict[str, list[str]] = {} + section_lines: dict[str, int] = {} + for line_no, line in enumerate(lines, 1): + if line.startswith("## "): + section_lines[line.removeprefix("## ").strip()] = line_no if line.startswith("| Area | Rows |"): header = [normalize_header(cell) for cell in split_cells(line)] - elif header is not None and line.startswith("| **Total** |"): - total = [cell.replace("*", "").strip() for cell in split_cells(line)] - break + elif header is not None and total is None and line.startswith("|"): + cells = [cell.replace("*", "").strip() for cell in split_cells(line)] + if is_separator(cells): + continue + if cells[0] == "Total": + total = cells + else: + summaries[cells[0]] = cells if header is None or total is None or len(header) != len(total): errors.append(f"{ENGINE_MATRIX.relative_to(ROOT)}: missing or malformed lifecycle summary") return actual_rows = [row for row in rows if row.path == ENGINE_MATRIX] - expected = {"rows": len(actual_rows)} - expected.update( - {normalize_header(state): sum(row.state == state for row in actual_rows) for state in STATES} - ) - for index, name in enumerate(header[1:], 1): - if name not in expected: - continue - try: - recorded = int(total[index]) - except ValueError: - errors.append(f"{ENGINE_MATRIX.relative_to(ROOT)}: non-numeric total for {name}") + + def check_counts(label: str, recorded_cells: list[str], scoped_rows: list[ClaimRow]) -> None: + if len(recorded_cells) != len(header): + errors.append(f"{ENGINE_MATRIX.relative_to(ROOT)}: malformed {label} lifecycle summary") + return + expected = {"rows": len(scoped_rows)} + expected.update( + {normalize_header(state): sum(row.state == state for row in scoped_rows) for state in STATES} + ) + for index, name in enumerate(header[1:], 1): + if name not in expected: + continue + try: + recorded = int(recorded_cells[index]) + except ValueError: + errors.append( + f"{ENGINE_MATRIX.relative_to(ROOT)}: non-numeric {label} summary for {name}" + ) + continue + if recorded != expected[name]: + errors.append( + f"{ENGINE_MATRIX.relative_to(ROOT)}: {label} summary {name}={recorded}; " + f"actual {expected[name]}" + ) + + check_counts("total", total, actual_rows) + for area, section in ENGINE_SUMMARY_SECTIONS: + recorded_cells = summaries.get(area) + section_line = section_lines.get(section) + if recorded_cells is None or section_line is None: + errors.append(f"{ENGINE_MATRIX.relative_to(ROOT)}: missing {area} lifecycle summary") continue - if recorded != expected[name]: - errors.append( - f"{ENGINE_MATRIX.relative_to(ROOT)}: summary {name}={recorded}; actual {expected[name]}" - ) + next_section_line = min( + (line_no for line_no in section_lines.values() if line_no > section_line), + default=len(lines) + 1, + ) + scoped_rows = [ + row for row in actual_rows if section_line < row.line_no < next_section_line + ] + check_counts(area, recorded_cells, scoped_rows) def is_placeholder(value: str) -> bool: diff --git a/scripts/check-pr-size.py b/scripts/check-pr-size.py index 5e7c6f7d..bc3f953c 100755 --- a/scripts/check-pr-size.py +++ b/scripts/check-pr-size.py @@ -92,6 +92,7 @@ PROCEDURE_FILES = frozenset( { "AGENTS.md", + "CLAUDE.md", ".agents/workflow.md", ".agents/verification.md", ".agents/porting.md", @@ -122,9 +123,11 @@ "scripts/agent-preflight.sh", } ) +PRODUCT_CHECKER_FILES = frozenset({"scripts/check-release-binary-contract.py"}) PUBLIC_DOCUMENT_FILES = frozenset( { "README.md", + "MANIFESTO.md", "CONTRIBUTING.md", "docs/STATUS.md", "docs/BENCHMARKS.md", @@ -144,8 +147,12 @@ HOOK = re.compile(r"\.githooks/(?:README\.md|[A-Za-z0-9_.-]+)\Z") BENCH_EVIDENCE = re.compile(r"(?:benchmarks/(?:demo|media)|docs/bench-evidence)/[A-Za-z0-9_.-]+\.(?:json|png|gif|mp4|log)\Z") ASSET = re.compile(r"assets/[A-Za-z0-9_.-]+\.(?:png|svg)\Z") +RELEASE_MANIFEST_FIXTURE = re.compile( + r"tests/scripts/fixtures/release_manifest/v[0-9]+/[a-z0-9-]+\.json\Z" +) CHECKER_EVIDENCE_OVERRIDES = { + "scripts/check-agent-record.py": "tests/scripts/test_agent_record.py", "scripts/check-policy.py": "tests/scripts/test_policy_contract.py", "scripts/check-role-discipline.py": "tests/scripts/test_check_pr_size.py", "scripts/check-doc-checkpoint.py": "tests/scripts/test_doc_checkpoint.py", @@ -248,6 +255,8 @@ def classify_path(path: str) -> str: return "design" if path in PUBLIC_DOCUMENT_FILES or DOC.fullmatch(path): return "public_document" + if path in PRODUCT_CHECKER_FILES: + return "product" if CHECKER.fullmatch(path): return "governance_checker" if CHECKER_TEST.fullmatch(path): @@ -256,10 +265,15 @@ def classify_path(path: str) -> str: return "ci" if HOOK.fullmatch(path): return "ci" - if ASSET.fullmatch(path): + if ASSET.fullmatch(path) or RELEASE_MANIFEST_FIXTURE.fullmatch(path): return "asset" if path.startswith("third_party/"): return "vendored_dependency" + if path in { + "release/manifest-v1.schema.json", + "scripts/env-doc-allowlist.txt", + }: + return "configuration" if path in { "CMakeLists.txt", ".env.example", ".gitignore", ".dockerignore", ".clang-format", ".gitattributes", "flake.lock", "flake.nix", diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index eb5657a1..52790711 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -401,7 +401,12 @@ def features_errors(text: str) -> list[str]: # ON the page deliberately: only OLMo-2 has a committed golden, Granite and # StableLM skip rather than gate, dflash and deepseek_v2 are build-verified # only. Nothing was lost, only de-duplicated. Net -2185. - "chars": 276960, + # + # 276860 since 2026-08-09 (measured 276835): the release-binary manifest + # milestone updated the OpenAI server status cell. Its endpoint-by-endpoint + # prose already lives in docs/USAGE.md, so the cell now keeps only the + # capability groups and binding limitations. Net -382 from current main. + "chars": 276860, "h2_sections": 11, "long_paragraphs": 89, "oversized_cells": 47, diff --git a/scripts/check-release-binary-contract.py b/scripts/check-release-binary-contract.py index 9aae4758..ab7ad0f8 100644 --- a/scripts/check-release-binary-contract.py +++ b/scripts/check-release-binary-contract.py @@ -55,91 +55,71 @@ ANCHORS = { ".agents/engine-matrix.md": "| `ENG-RELEASE-BINARIES` |", ".agents/roadmap_v1.md": "| REL | `ROAD-V1-RELEASE` |", - ".agents/NOW.md": "| Release | SPIKE; 30/30 | #129 |", - ".agents/coordination.md": ( - "| `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | " - "`ENG-RELEASE-BINARIES` |" - ), - ".agents/state.md": ( - "## 2026-08-07 — Release matrix revised: fat CUDA and adaptive CPU " - "are the primary downloads" - ), - "docs/STATUS.md": ( - "#129: SPIKE∅" - ), - "docs/BENCHMARKS.md": ( - "| **Binary release matrix (spiked)** | `ENG-RELEASE-BINARIES`:" - ), + ".agents/NOW.md": "| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** |", + ".agents/coordination.md": "| `CLAIM-ENG-RELEASE-BINARIES-W5` | `ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) |", + ".agents/state.md": "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE", + "docs/STATUS.md": "#141 W5 19/19 ACTIVE ART∅", + "docs/BENCHMARKS.md": "| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`:", } LIFECYCLE_RECORD_MUTATIONS = ( ( ".agents/engine-matrix.md", - "`SPIKE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` |", - "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` |", + "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", + "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", "engine-matrix release lifecycle", ), ( ".agents/engine-matrix.md", - "gaps remain; no install/archive/publish implementation", - "gaps closed; install/archive/publish implementation complete", + "install/archive/publish implementation and all real release evidence remain pending", + "install/archive/publish implementation and all real release evidence are complete", "engine-matrix release lifecycle", ), ( ".agents/roadmap_v1.md", - "`SPIKE` | Fresh review of PR #129", - "`DONE` | Fresh review of PR #129", + "`ACTIVE` | W5 versioned manifest schema/generator", + "`DONE` | W5 versioned manifest schema/generator", "roadmap release lifecycle", ), ( ".agents/roadmap_v1.md", - "bundle work; no archive exists", - "bundle work complete; archive exists", + "no archive exists", + "archive exists", "roadmap release lifecycle", ), ( ".agents/coordination.md", - "| `ACTIVE` | 2026-08-07 — user-reviewed revision complete: primary fat " - "CUDA + adaptive CPU per host ABI, optional per-SM diagnostics; row stays " - "`SPIKE`; awaiting fresh review |", - "| `DONE` | 2026-08-07 — user-reviewed revision complete: primary fat " - "CUDA + adaptive CPU per host ABI, optional per-SM diagnostics; row stays " - "`SPIKE`; awaiting fresh review |", + "| `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; no archive or real runtime/correctness/performance evidence |", + "| `DONE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; archive and runtime/correctness/performance evidence complete |", "coordination release lifecycle", ), ( ".agents/coordination.md", - "no CMake, workflow, source, test, or artifact implementation", - "CMake, workflow, source, test, and artifact implementation complete", + "Excludes W1-W4, W6-W13, archives, install/package/publish workflows and runtime artifacts", + "Includes W1-W13, archives, install/package/publish workflows and runtime artifacts", "coordination release lifecycle", ), ( ".agents/coordination.md", - "row stays `SPIKE`; awaiting fresh review", - "row is `DONE`; release shipped", + "no archive or real runtime/correctness/performance evidence", + "release archive and runtime/correctness/performance evidence shipped", "coordination release lifecycle", ), ( ".agents/state.md", - "`ENG-RELEASE-BINARIES` remains `SPIKE`, and no archive or implementation " - "is\nclaimed.", - "`ENG-RELEASE-BINARIES` is `DONE`, with archive and implementation.", + "`ACTIVE`, never `DONE` or `GATING`.", + "`DONE`, with every release gate complete.", "state release lifecycle", ), ) BENCHMARKS_RELEASE_ROW = ( - "| **Binary release matrix (spiked)** | `ENG-RELEASE-BINARIES`: primary " - "host-ABI fat-CUDA + adaptive-CPU static-core bundles; optional per-SM " - "diagnostics; experimental literal-static musl CPU | **PENDING:** pins 10-SM " - "fat CUDA, adaptive no-AVX2 CPU, W1-W13/W10-W12 policy, public pending states; " - "30 tests GREEN. No archive, staged smoke, runtime, correctness, or performance " - "evidence " - "| n/a |" + "| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`: versioned deterministic manifest for primary host-ABI fat-CUDA + adaptive-CPU static-core bundles, optional per-SM diagnostics and experimental literal-static musl CPU | **W5 GREEN:** " + "19/19 + ten review mutations killed; contract 30/30; PR-size classes/budget GREEN. Synthetic fixtures only. **PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates | n/a |" ) STATUS_RELEASE_FRAGMENTS = ( - "Supported (subset); #129: SPIKE∅", + "Subset; #141 W5 19/19 ACTIVE ART∅", ) BACKEND_POLICY_PROSE = { @@ -293,17 +273,14 @@ PUBLIC_PENDING_MUTATIONS = ( ( "docs/BENCHMARKS.md", - "**PENDING:** pins 10-SM fat CUDA, adaptive no-AVX2 CPU, " - "W1-W13/W10-W12 policy, public pending states; 30 tests GREEN. No archive, " - "staged smoke, runtime, correctness, or performance evidence", - "**SHIPPED:** archive, runtime, correctness, and performance evidence " - "complete", + "**PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates", + "**SHIPPED:** archive, runtime, correctness, and performance evidence complete", "docs/BENCHMARKS.md release row", ), ( "docs/STATUS.md", - "Supported (subset); #129: SPIKE∅", - "Supported; #129: SHIPPED", + "Subset; #141 W5 19/19 ACTIVE ART∅", + "Supported; #141: RELEASE DONE/ARTIFACTS✓", "docs/STATUS.md release row", ), ) @@ -421,7 +398,9 @@ TEST_LITERAL_INVENTORIES = { "PRIMARY_CUDA_SMS": PRIMARY_CUDA_SMS, "EXACT_MACHINE_FIELDS": { - "lifecycle": "SPIKE", + "lifecycle": "ACTIVE", + "manifest_schema": "vllm.cpp.release-manifest.v1", + "work_W5_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -435,7 +414,8 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/state.md,docs/STATUS.md," - "docs/BENCHMARKS.md" + "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" ), }, "EXPECTED_DEPS": {work: ",".join(deps) for work, deps in WORK_DEPS.items()}, @@ -578,7 +558,7 @@ "GUARD_MAP_KEYS": "701e4821bee926c2e074dbf2b97ff4a93bebb610cc6bed76e06063cab8974758", "INVENTORY_CONSUMER_METHODS": "916894a32d88026a883cc1f316d949eb116ee1fced36d635b585d7bf3372b01d", "CONSUMER_FLOW_MUTATIONS": "6f69f9e361d38c325fbc455c31ec7211578131368624312e024448afdfc01e83", - "UNKNOWN_MACHINE_FIELD_MUTATIONS": "8d10128593c67c64cde8cbe7e58faa9a9ade0d4bde391d24474fd04d6392ed6b", + "UNKNOWN_MACHINE_FIELD_MUTATIONS": "b69a6bd26c8417e04994815042ba1520968b906d6bfee4b3413ffc0dafafc5f2", "HUMAN_WORK_DEPS": "54a501b903eb3c97023084393666f9f63d289ab9a78e22f389c32bfc1711573b", "BACKEND_POLICY_PROSE_MUTATIONS": "c5fea18a668932c4768cb9feb4746fd444b3df7e7ec15df1a588141898d28f2d", "PREFLIGHT_WIRING_MUTATIONS": "d442c6d188efd624bffc9e94a7750d6a527c7b693affde5cbc33304f9e95272e", @@ -586,7 +566,9 @@ } EXACT_MACHINE_FIELDS = { - "lifecycle": "SPIKE", + "lifecycle": "ACTIVE", + "manifest_schema": "vllm.cpp.release-manifest.v1", + "work_W5_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -600,7 +582,8 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/state.md,docs/STATUS.md," - "docs/BENCHMARKS.md" + "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" ), } @@ -621,12 +604,10 @@ ) STATE_RELEASE_HEADING = ( - "## 2026-08-07 — Release matrix revised: fat CUDA and adaptive CPU are the " - "primary downloads" + "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE" ) STATE_RELEASE_LIFECYCLE = ( - "`ENG-RELEASE-BINARIES` remains `SPIKE`, and no archive or implementation " - "is claimed." + "The row advances from `SPIKE` to `ACTIVE`, never `DONE` or `GATING`." ) @@ -947,13 +928,18 @@ def wiring_errors(preflight_text: str, ci_text: str) -> list[str]: errors.append("release checker is missing from preflight CHECKERS") if suites is None or "test_check_release_binary_contract" not in suites: errors.append("release mutation suite is missing from preflight SUITES") + if suites is None or "test_release_manifest" not in suites: + errors.append("W5 manifest suite is missing from preflight SUITES") returncode, invocations = _trace_preflight_commands(preflight_text) checker_argv = ("scripts/check-release-binary-contract.py",) suite_argv = ("tests/scripts/test_check_release_binary_contract.py",) + manifest_suite_argv = ("tests/scripts/test_release_manifest.py",) if invocations.count(checker_argv) != 1: errors.append("preflight does not execute release CHECKERS through its checker loop") if invocations.count(suite_argv) != 1: errors.append("preflight does not execute release SUITES through its suite loop") + if invocations.count(manifest_suite_argv) != 1: + errors.append("preflight does not execute the W5 manifest suite exactly once") if returncode != 0: errors.append(f"instrumented preflight execution failed with rc={returncode}") active = _active_ci_commands(ci_text) @@ -964,6 +950,8 @@ def wiring_errors(preflight_text: str, ci_text: str) -> list[str]: "tests/scripts/test_check_release_binary_contract.py", ) not in active: errors.append("release mutation suite is missing from the explicit CI step") + if ("python3", "tests/scripts/test_release_manifest.py") not in active: + errors.append("W5 manifest suite is missing from an unconditional CI step") if not _ci_has_active_release_step(ci_text): errors.append( "CI release step must contain checker and suite as direct active commands" @@ -1014,13 +1002,9 @@ def _release_lifecycle_errors(root: Path) -> list[str]: "engine-matrix release lifecycle", errors, ) - if engine is not None and ( - engine[7] != "`SPIKE`" - or "gaps remain; no install/archive/publish implementation" not in engine[4] - ): + if engine is not None and (engine[7] != "`ACTIVE`" or "W5 schema" not in engine[4] or "install/archive/publish implementation and all real release evidence remain pending" not in engine[4]): errors.append( - "engine-matrix release lifecycle must stay SPIKE and state that the " - "gaps remain with no install/archive/publish implementation" + "engine-matrix release lifecycle must be ACTIVE with W5 implemented and install/archive/publish plus real evidence pending" ) roadmap = _table_record( @@ -1031,31 +1015,22 @@ def _release_lifecycle_errors(root: Path) -> list[str]: "roadmap release lifecycle", errors, ) - if roadmap is not None and ( - roadmap[5] != "`SPIKE`" or "no archive exists" not in roadmap[6] - ): + if roadmap is not None and (roadmap[5] != "`ACTIVE`" or "W5 versioned manifest" not in roadmap[6] or "W1-W4 and W6-W13 remain pending" not in roadmap[6] or "no archive exists" not in roadmap[6]): errors.append( - "roadmap release lifecycle must stay SPIKE and state that no archive " - "exists" + "roadmap release lifecycle must be ACTIVE with W5 implemented while W1-W4/W6-W13 and archives remain pending" ) coordination = _table_record( root, ".agents/coordination.md", - "| `CLAIM-ENG-RELEASE-BINARIES-SPIKE` |", + "| `CLAIM-ENG-RELEASE-BINARIES-W5` |", 8, "coordination release lifecycle", errors, ) - if coordination is not None and ( - coordination[6] != "`ACTIVE`" - or "no CMake, workflow, source, test, or artifact implementation" - not in coordination[5] - or "row stays `SPIKE`; awaiting fresh review" not in coordination[7] - ): + if coordination is not None and (coordination[6] != "`ACTIVE`" or "W5 implemented" not in coordination[5] or "Excludes W1-W4, W6-W13" not in coordination[5] or "no archive or real runtime/correctness/performance evidence" not in coordination[7]): errors.append( - "coordination release lifecycle must keep the claim ACTIVE while the " - "row stays SPIKE, awaiting review, with no implementation" + "coordination release lifecycle must keep W5 ACTIVE and implemented while later work and real artifact evidence remain pending" ) state_path = root / ".agents/state.md" @@ -1075,10 +1050,7 @@ def _release_lifecycle_errors(root: Path) -> list[str]: end = state_text.find("\n## ", start) section = state_text[start:] if end < 0 else state_text[start:end] if STATE_RELEASE_LIFECYCLE not in _normalize_prose(section): - errors.append( - "state release lifecycle must say ENG-RELEASE-BINARIES remains " - "SPIKE with no archive or implementation claimed" - ) + errors.append("state release lifecycle must say W5 advances the row only to ACTIVE, never DONE or GATING") return errors @@ -1283,10 +1255,9 @@ def contract_errors(root: Path) -> list[str]: if fields.get(key) != expected: errors.append(_field_error(key, fields.get(key), expected)) - if "Status: accepted spike for `ENG-RELEASE-BINARIES`." not in text: + if "Status: accepted contract with W5 manifest tooling implemented for\n`ENG-RELEASE-BINARIES`." not in text: errors.append( - "release spec identity/status line must name accepted spike " - "ENG-RELEASE-BINARIES" + "release spec identity/status line must name the accepted W5 ENG-RELEASE-BINARIES contract" ) header = "| Work | Deps | Deliverable | Exit gate |" @@ -1363,7 +1334,7 @@ def contract_errors(root: Path) -> list[str]: ) if not all(fragment in status_row for fragment in STATUS_RELEASE_FRAGMENTS): errors.append( - "docs/STATUS.md release row must stay SPIKED with no artifacts and no " + "docs/STATUS.md release row must stay ACTIVE/W5 with no artifacts or " "runtime claim" ) preflight = root / PREFLIGHT_PATH diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 9a76e0f3..af2a1dc6 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -97,6 +97,13 @@ VT_GDN_WY_BLOCKED VT_GEMMA2_ATTN_SOFTCAP VT_GEMMA2_SLIDING VT_GEMMA3_SLIDING +VT_GEMMA4_BATCH_EXPERTS +VT_GEMMA4_CUSTOM_EXPERT +VT_GEMMA4_EXPERT_VRAM_MB +VT_GEMMA4_FP8_NATIVE +VT_GEMMA4_FUSED_EXPERTS +VT_GEMMA4_HOST_AXPY +VT_GEMMA4_PROFILE VT_GEMM_ALGO_LOG VT_GGUF_GDN_NK VT_GGUF_NO_SPLIT @@ -134,6 +141,9 @@ VT_QWEN3_ROPE_CACHE VT_RELEASE_HOST_WEIGHTS VT_RMSNORM_DECODE_FAST VT_RMSNORM_GATED_FAST +VT_ROCM_GEMM_COMPUTE +VT_ROCM_GEMV +VT_ROCM_HIPBLASLT VT_SILU_FP4_FAST VT_SWIZZLE_IN_QUANT VT_W4A4_TRUE diff --git a/scripts/release_manifest.py b/scripts/release_manifest.py new file mode 100644 index 00000000..e0406b2e --- /dev/null +++ b/scripts/release_manifest.py @@ -0,0 +1,753 @@ +#!/usr/bin/env python3 +"""Generate and validate vllm.cpp release manifests using only the stdlib. + +This is build-time tooling. Release archives contain the generated JSON, not +this Python program or a Python runtime. CUDA feature resolution is derived from +the same VT_CUDA_FEATURE_TABLE consumed by CMake, so the manifest cannot invent +a capability independently of the build definition. +""" + +from __future__ import annotations + +import argparse +import copy +import json +import re +import sys +from pathlib import Path +from typing import Any + + +SCHEMA_ID = "vllm.cpp.release-manifest.v1" +SCHEMA_VERSION = 1 +DEFAULT_SCHEMA = Path("release/manifest-v1.schema.json") +FEATURE_TABLE = Path("cmake/CudaArchFeatures.cmake") + +PRIMARY_CUDA_SMS = ( + "80", "86", "87", "89", "90a", "100a", "103a", "110", "120a", "121a" +) +AOT_AVAILABILITY = { + "80": True, "86": True, "87": False, "89": True, "90a": True, + "100a": True, "103a": False, "110": False, "120a": False, "121a": True, +} +PUBLISHED_EVIDENCE = ("build", "archive_smoke", "dependency_audit") +STABLE_EVIDENCE = ("runtime", "correctness") +STABLE_SUPPLY_CHAIN_EVIDENCE = ( + "archive_checksum", "sbom", "provenance", "licenses" +) + +# Current compiled CPU inventory. This is intentionally narrower than the ISA +# names a compiler understands: a tier is listed only when a real kernel TU and +# runtime selector exist. x86 anchors: CMakeLists.txt:912-923 and +# src/vt/cpu/cpu_matmul_elem.cpp:533-612; Arm anchors: CMakeLists.txt:926-933, +# src/vt/cpu/cpu_quant_dot_arm.cpp:1-40,67-83 and +# src/vt/cpu/cpu_quant_repack_arm.cpp:1-22,52-68. +CPU_TIER_POLICY = { + "x86_64": { + "baseline": "portable-sse2", + "tiers": ( + "portable-sse2", "sse2-f16c", "avx2-f16c", "avx512f" + ), + "kernel_families": { + "portable-sse2": {"matmul-elem-f32-bf16"}, + "sse2-f16c": {"matmul-elem-f16"}, + "avx2-f16c": {"matmul-elem-f32-bf16-f16"}, + "avx512f": {"matmul-elem-f32-bf16"}, + }, + "bits": { + "portable-sse2": {"sse2"}, + "sse2-f16c": {"sse2", "avx", "f16c", "osxsave"}, + "avx2-f16c": {"avx", "avx2", "f16c", "osxsave"}, + "avx512f": {"avx", "avx512f", "osxsave"}, + }, + "os_state": { + "portable-sse2": set(), + "sse2-f16c": {"xcr0:xmm", "xcr0:ymm"}, + "avx2-f16c": {"xcr0:xmm", "xcr0:ymm"}, + "avx512f": { + "xcr0:xmm", "xcr0:ymm", "xcr0:opmask", + "xcr0:zmm_hi256", "xcr0:hi16_zmm", + }, + }, + }, + "aarch64": { + "baseline": "portable-neon", + "tiers": ("portable-neon", "i8mm"), + "kernel_families": { + "portable-neon": {"matmul-elem-f32-bf16-f16"}, + "i8mm": { + "quant-dot-q4_0-q8_0-q4_K-q6_K", + "quant-repack-q8_0", + }, + }, + "bits": {"portable-neon": {"neon"}, "i8mm": {"i8mm"}}, + "os_state": { + "portable-neon": {"linux": set(), "macos": set()}, + "i8mm": { + "linux": {"getauxval:AT_HWCAP2:HWCAP2_I8MM"}, + "macos": {"sysctl:hw.optional.arm.FEAT_I8MM"}, + }, + }, + }, +} + + +class ManifestError(ValueError): + """Raised when generation input cannot produce a valid manifest.""" + + +def canonical_json(value: Any) -> str: + return json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n" + + +def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + value: dict[str, Any] = {} + for key, item in pairs: + if key in value: + raise ManifestError(f"duplicate JSON key {key!r}") + value[key] = item + return value + + +def load_schema(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as handle: + value = json.load(handle, object_pairs_hook=_reject_duplicate_keys) + if not isinstance(value, dict): + raise ManifestError(f"schema {path} must be a JSON object") + return value + + +def _resolve_ref(ref: str, root: dict[str, Any]) -> dict[str, Any]: + if not ref.startswith("#/"): + raise ManifestError(f"unsupported non-local schema reference {ref!r}") + value: Any = root + for component in ref[2:].split("/"): + component = component.replace("~1", "/").replace("~0", "~") + value = value[component] + if not isinstance(value, dict): + raise ManifestError(f"schema reference {ref!r} is not an object") + return value + + +def _type_matches(value: Any, expected: str) -> bool: + if expected == "object": + return isinstance(value, dict) + if expected == "array": + return isinstance(value, list) + if expected == "string": + return isinstance(value, str) + if expected == "boolean": + return isinstance(value, bool) + if expected == "integer": + return isinstance(value, int) and not isinstance(value, bool) + raise ManifestError(f"unsupported schema type {expected!r}") + + +def _json_equal(left: Any, right: Any) -> bool: + """Compare JSON values without Python's bool/int equality collapse.""" + if type(left) is not type(right): + return False + return left == right + + +def _schema_errors( + value: Any, + schema: dict[str, Any], + root: dict[str, Any], + path: str = "$", +) -> list[str]: + if "$ref" in schema: + return _schema_errors(value, _resolve_ref(schema["$ref"], root), root, path) + errors: list[str] = [] + expected_type = schema.get("type") + if expected_type is not None and not _type_matches(value, expected_type): + return [f"{path}: expected {expected_type}, got {type(value).__name__}"] + if "const" in schema and not _json_equal(value, schema["const"]): + errors.append(f"{path}: must equal {schema['const']!r}") + if "enum" in schema and not any( + _json_equal(value, candidate) for candidate in schema["enum"] + ): + errors.append(f"{path}: {value!r} is not one of {schema['enum']!r}") + if isinstance(value, str) and len(value) < schema.get("minLength", 0): + errors.append(f"{path}: string is shorter than minLength") + if isinstance(value, dict): + properties = schema.get("properties", {}) + for key in schema.get("required", []): + if key not in value: + errors.append(f"{path}: required property {key!r} is missing") + if schema.get("additionalProperties") is False: + for key in value: + if key not in properties: + errors.append(f"{path}: unknown property {key!r}") + for key, child in value.items(): + if key in properties: + errors.extend( + _schema_errors(child, properties[key], root, f"{path}.{key}") + ) + if isinstance(value, list): + if len(value) < schema.get("minItems", 0): + errors.append(f"{path}: array has fewer than minItems entries") + if schema.get("uniqueItems"): + encoded = [json.dumps(item, sort_keys=True) for item in value] + if len(encoded) != len(set(encoded)): + errors.append(f"{path}: array entries must be unique") + item_schema = schema.get("items") + if item_schema: + for index, item in enumerate(value): + errors.extend( + _schema_errors(item, item_schema, root, f"{path}[{index}]") + ) + return errors + + +def _release_arch(value: str) -> str: + value = value.replace("+PTX", "").strip() + match = re.fullmatch(r"(\d+)\.(\d+)(a?)", value) + if match: + return f"{match.group(1)}{match.group(2)}{match.group(3)}" + if re.fullmatch(r"\d+(?:a)?", value): + return value + raise ManifestError(f"cannot normalize CUDA architecture {value!r}") + + +def parse_cuda_feature_table(repo_root: Path) -> list[tuple[str, tuple[str, ...]]]: + path = repo_root / FEATURE_TABLE + text = path.read_text(encoding="utf-8") + rows: list[tuple[str, tuple[str, ...]]] = [] + in_table = False + for line in text.splitlines(): + if line.strip() == "set(VT_CUDA_FEATURE_TABLE": + in_table = True + continue + if in_table and line.strip() == ")": + break + if not in_table: + continue + match = re.match(r'^\s*"([^|]+)\|([^|]*)\|', line) + if not match: + continue + arches = tuple(_release_arch(item) for item in match.group(2).split(",") if item) + rows.append((match.group(1), arches)) + if not rows: + raise ManifestError(f"{path}: VT_CUDA_FEATURE_TABLE is empty or unreadable") + names = [name for name, _ in rows] + if len(names) != len(set(names)): + raise ManifestError(f"{path}: duplicate CUDA feature name") + return rows + + +def resolve_cuda_features( + compiled_sms: list[str], repo_root: Path +) -> list[dict[str, Any]]: + requested = set(compiled_sms) + return [ + {"name": name, "compiled_sms": [sm for sm in compiled_sms if sm in arches]} + for name, arches in parse_cuda_feature_table(repo_root) + ] + + +def _evidence_objects(manifest: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]: + objects: list[tuple[str, dict[str, Any]]] = [] + evidence_set = manifest.get("evidence") + if isinstance(evidence_set, dict): + for key, value in evidence_set.items(): + if isinstance(value, dict): + objects.append((f"$.evidence.{key}", value)) + supply_chain = manifest.get("supply_chain") + if isinstance(supply_chain, dict): + for key, value in supply_chain.items(): + if isinstance(value, dict): + objects.append((f"$.supply_chain.{key}", value)) + cpu = manifest.get("cpu") + if isinstance(cpu, dict): + for index, tier in enumerate(cpu.get("compiled_tiers", [])): + if isinstance(tier, dict) and isinstance(tier.get("execution_evidence"), dict): + objects.append( + (f"$.cpu.compiled_tiers[{index}].execution_evidence", tier["execution_evidence"]) + ) + cuda = manifest.get("cuda") + if isinstance(cuda, dict): + for index, sm in enumerate(cuda.get("sm_evidence", [])): + if not isinstance(sm, dict): + continue + for key in ("aot_evidence", "runtime_evidence"): + if isinstance(sm.get(key), dict): + objects.append((f"$.cuda.sm_evidence[{index}].{key}", sm[key])) + return objects + + +def _validate_evidence(path: str, value: dict[str, Any]) -> list[str]: + state = value.get("state") + reason = value.get("reason") + command = value.get("command") + result = value.get("result") + url = value.get("url") + if not all(isinstance(item, str) for item in (reason, command, result, url)): + return [] # the schema reports the type error + errors: list[str] = [] + if state == "passed": + if reason: + errors.append(f"{path}: passed evidence reason must be empty") + for field, content in (("command", command), ("result", result), ("url", url)): + if not content: + errors.append(f"{path}: passed evidence requires non-empty {field}") + elif state == "failed": + for field, content in (("reason", reason), ("command", command), ("result", result)): + if not content: + errors.append(f"{path}: failed evidence requires non-empty {field}") + elif state in {"absent", "not-applicable"}: + if not reason: + errors.append(f"{path}: {state} evidence requires a reason") + for field, content in (("command", command), ("result", result), ("url", url)): + if content: + errors.append(f"{path}: {state} evidence must leave {field} empty") + return errors + + +def _artifact_policy(manifest: dict[str, Any]) -> list[str]: + artifact = manifest.get("artifact", {}) + host = manifest.get("host", {}) + backend = manifest.get("backend", {}) + if not all(isinstance(value, dict) for value in (artifact, host, backend)): + return [] + artifact_id = artifact.get("id") + name = backend.get("name") + policies = { + "linux-x86_64-glibc-cpu": ("linux", "x86_64", "glibc", "cpu", "static-core", {"preview", "stable"}), + "linux-aarch64-glibc-cpu": ("linux", "aarch64", "glibc", "cpu", "static-core", {"preview", "stable"}), + "linux-x86_64-glibc-cuda-fat": ("linux", "x86_64", "glibc", "cuda", "static-core", {"preview", "stable"}), + "linux-aarch64-glibc-cuda-fat": ("linux", "aarch64", "glibc", "cuda", "static-core", {"preview", "stable"}), + "macos-arm64-metal": ("macos", "aarch64", "macos", "metal", "static-core", {"preview", "stable"}), + "macos-arm64-metal-mlx": ("macos", "aarch64", "macos", "mlx", "static-core", {"preview"}), + "linux-x86_64-glibc-vulkan": ("linux", "x86_64", "glibc", "vulkan", "static-core", {"preview"}), + "linux-x86_64-musl-cpu-static": ("linux", "x86_64", "musl", "cpu", "literal-static", {"experimental-preview"}), + } + policy = policies.get(artifact_id) + if policy is None and artifact.get("kind") == "diagnostic" and name == "cuda": + if not isinstance(artifact_id, str) or not re.fullmatch( + r"linux-(?:x86_64|aarch64)-glibc-cuda-sm(?:80|86|87|89|90a|100a|103a|110|120a|121a)", + artifact_id, + ): + return ["$.artifact.id: unknown diagnostic CUDA artifact tuple"] + return [] + if policy is None: + return ["$.artifact.id: unknown release artifact tuple"] + os_name, arch, abi, expected_backend, static_boundary, channels = policy + errors: list[str] = [] + actual = (host.get("os"), host.get("arch"), host.get("abi"), name, artifact.get("static_boundary")) + expected = (os_name, arch, abi, expected_backend, static_boundary) + if actual != expected: + errors.append(f"$.artifact: tuple policy mismatch; expected {expected!r}, got {actual!r}") + if artifact.get("channel") not in channels: + errors.append(f"$.artifact.channel: wrong channel for {artifact_id}") + if artifact.get("kind") != "primary": + errors.append(f"$.artifact.kind: matrix artifact {artifact_id} must be primary") + return errors + + +def _backend_policy(manifest: dict[str, Any], repo_root: Path) -> list[str]: + backend = manifest.get("backend", {}) + if not isinstance(backend, dict): + return [] + name = backend.get("name") + flags = backend.get("flags", {}) + if not isinstance(flags, dict): + return [] + expected_switches = { + "VLLM_CPP_CUDA": name == "cuda", + "VLLM_CPP_HIP": False, + "VLLM_CPP_METAL": name in {"metal", "mlx"}, + "VLLM_CPP_MLX": name == "mlx", + "VLLM_CPP_VULKAN": name == "vulkan", + } + errors: list[str] = [] + for flag, expected in expected_switches.items(): + if flags.get(flag) is not expected: + errors.append(f"$.backend.flags.{flag}: inconsistent backend flags for {name}") + if flags.get("VLLM_CPP_SERVER") is not True or flags.get("VLLM_CPP_BUILD_EXAMPLES") is not True: + errors.append("$.backend.flags: release backend flags must build server and examples") + if flags.get("VLLM_CPP_HIP_ARCHITECTURES") != []: + errors.append("$.backend.flags.VLLM_CPP_HIP_ARCHITECTURES: HIP is blocked") + if name == "cuda": + if flags.get("VLLM_CPP_TRITON") is not True: + errors.append("$.backend.flags.VLLM_CPP_TRITON: CUDA release input must resolve Triton explicitly") + compiled = manifest.get("cuda", {}).get("compiled_sms", []) if isinstance(manifest.get("cuda"), dict) else [] + expected_features = resolve_cuda_features(compiled, repo_root) + if backend.get("resolved_features") != expected_features: + errors.append("$.backend.resolved_features: does not match resolved CUDA feature table") + if flags.get("VLLM_CPP_CUDA_ARCHITECTURES") != compiled: + errors.append("$.backend.flags.VLLM_CPP_CUDA_ARCHITECTURES: must equal compiled CUDA SMs") + if backend.get("gpu_driver_boundary") != "external-host-never-bundled": + errors.append("$.backend.gpu_driver_boundary: CUDA driver boundary must stay external") + else: + if flags.get("VLLM_CPP_CUDA_ARCHITECTURES") != []: + errors.append("$.backend.flags.VLLM_CPP_CUDA_ARCHITECTURES: non-CUDA artifact cannot claim SMs") + if backend.get("resolved_features") != []: + errors.append("$.backend.resolved_features: non-CUDA artifact cannot claim CUDA features") + if name == "cpu" and flags.get("VLLM_CPP_TRITON") is not False: + errors.append("$.backend.flags.VLLM_CPP_TRITON: CPU artifact cannot enable Triton") + if name == "mlx": + if not flags.get("MLX_ROOT"): + errors.append("$.backend.flags.MLX_ROOT: MLX preview requires an explicit root") + elif flags.get("MLX_ROOT") != "": + errors.append("$.backend.flags.MLX_ROOT: only MLX artifacts may set MLX_ROOT") + if name in {"cuda", "metal", "mlx", "vulkan"}: + if backend.get("gpu_driver_boundary") != "external-host-never-bundled": + errors.append("$.backend.gpu_driver_boundary: accelerator driver boundary must stay external") + elif backend.get("gpu_driver_boundary") != "not-applicable": + errors.append("$.backend.gpu_driver_boundary: CPU driver boundary must be not-applicable") + return errors + + +def _dependency_policy(manifest: dict[str, Any]) -> list[str]: + dependencies = manifest.get("dependencies") + if not isinstance(dependencies, list): + return [] + backend = manifest.get("backend", {}) + artifact = manifest.get("artifact", {}) + supply_chain = manifest.get("supply_chain", {}) + backend_name = backend.get("name") if isinstance(backend, dict) else None + literal_static = ( + isinstance(artifact, dict) + and artifact.get("static_boundary") == "literal-static" + ) + errors: list[str] = [] + names: list[Any] = [] + for index, dependency in enumerate(dependencies): + if not isinstance(dependency, dict): + continue + dependency_name = dependency.get("name") + names.append(dependency_name) + if literal_static and dependency.get("role") != "build-time": + if ( + dependency.get("kind") == "driver" + or dependency.get("linkage") != "static" + or dependency.get("bundled") is not True + or dependency.get("role") == "external-runtime" + ): + errors.append( + f"$.dependencies[{index}]: literal-static runtime dependency " + "must be a bundled static non-driver" + ) + if backend_name == "cpu" and dependency.get("kind") == "driver": + errors.append(f"$.dependencies[{index}]: CPU artifact cannot declare a GPU driver") + if dependency.get("kind") == "driver": + if dependency.get("bundled") is not False: + errors.append(f"$.dependencies[{index}]: GPU driver must never be bundled") + if dependency.get("linkage") != "external" or dependency.get("role") != "external-runtime": + errors.append(f"$.dependencies[{index}]: GPU driver must be an external runtime boundary") + if dependency.get("bundled") and dependency.get("linkage") == "dynamic": + allowed_mlx = backend_name == "mlx" and dependency_name in { + "libmlx.dylib", "mlx.metallib" + } + if not allowed_mlx: + errors.append( + f"$.dependencies[{index}]: bundled dynamic dependency is not permitted" + ) + if len(names) != len(set(names)): + errors.append("$.dependencies: dependency names must be unique") + + by_name = { + dependency.get("name"): dependency + for dependency in dependencies + if isinstance(dependency, dict) + } + + def require_external(name: str, kind: str) -> None: + dependency = by_name.get(name) + if not isinstance(dependency, dict) or ( + dependency.get("kind") != kind + or dependency.get("linkage") != "external" + or dependency.get("bundled") is not False + or dependency.get("role") != "external-runtime" + ): + errors.append( + f"$.dependencies: {backend_name} requires external {name} declaration" + ) + + if backend_name == "cuda": + require_external("nvidia-driver", "driver") + for index, dependency in enumerate(dependencies): + if ( + isinstance(dependency, dict) + and dependency.get("kind") == "driver" + and dependency.get("name") != "nvidia-driver" + ): + errors.append( + f"$.dependencies[{index}]: CUDA accepts only the NVIDIA driver boundary" + ) + elif backend_name == "vulkan": + require_external("vulkan-loader", "library") + require_external("vulkan-icd", "library") + require_external("vulkan-driver", "driver") + elif backend_name in {"metal", "mlx"}: + require_external("Metal.framework", "framework") + require_external("Foundation.framework", "framework") + + if backend_name == "mlx": + for name in ("libmlx.dylib", "mlx.metallib"): + dependency = by_name.get(name) + if not isinstance(dependency, dict) or ( + dependency.get("kind") != "library" + or dependency.get("linkage") != "dynamic" + or dependency.get("bundled") is not True + or dependency.get("role") != "runtime" + or dependency.get("version") in {"", "unknown", "unversioned"} + ): + errors.append(f"$.dependencies: MLX requires bundled versioned {name}") + licenses = supply_chain.get("licenses", {}) if isinstance(supply_chain, dict) else {} + if not isinstance(licenses, dict) or licenses.get("state") != "passed": + errors.append("$.dependencies: bundled MLX dylib/metallib require passed license evidence") + return errors + + +def _build_policy(manifest: dict[str, Any]) -> list[str]: + build = manifest.get("build") + backend = manifest.get("backend") + if not isinstance(build, dict) or not isinstance(backend, dict): + return [] + errors: list[str] = [] + commit = build.get("source_commit") + if not isinstance(commit, str) or re.fullmatch(r"[0-9a-f]{40}", commit) is None: + errors.append("$.build.source_commit: must be a full lowercase 40-hex commit") + if build.get("resolved_cmake_options") != backend.get("flags"): + errors.append("$.build.resolved_cmake_options: must equal resolved CMake backend flags") + commands = build.get("test_commands") + if isinstance(commands, list) and len(commands) != len(set(commands)): + errors.append("$.build.test_commands: commands must be unique") + return errors + + +def _cpu_policy(manifest: dict[str, Any]) -> list[str]: + backend = manifest.get("backend", {}) + host = manifest.get("host", {}) + cpu = manifest.get("cpu") + backend_name = backend.get("name") if isinstance(backend, dict) else None + if backend_name != "cpu": + return ["$.cpu: only CPU artifacts may carry compiled CPU tiers"] if cpu is not None else [] + if not isinstance(cpu, dict) or not isinstance(host, dict): + return ["$.cpu: CPU artifact requires compiled CPU tiers"] + policy = CPU_TIER_POLICY.get(host.get("arch")) + if policy is None: + return ["$.host.arch: no compiled CPU tier inventory for host architecture"] + errors: list[str] = [] + if cpu.get("baseline") != policy["baseline"]: + errors.append(f"$.cpu.baseline: expected {policy['baseline']!r}") + tiers = cpu.get("compiled_tiers", []) + if not isinstance(tiers, list): + return errors + names = [tier.get("name") for tier in tiers if isinstance(tier, dict)] + if names != list(policy["tiers"]): + errors.append(f"$.cpu.compiled_tiers: compiled tiers must equal {list(policy['tiers'])!r}") + if cpu.get("selected_tier") not in names: + errors.append("$.cpu.selected_tier: selected tier must be one of the compiled tiers") + for index, tier in enumerate(tiers): + if not isinstance(tier, dict) or tier.get("name") not in policy["tiers"]: + continue + name = tier["name"] + kernel_families = set(tier.get("kernel_families", [])) + expected_kernel_families = set(policy["kernel_families"][name]) + if kernel_families != expected_kernel_families: + errors.append( + f"$.cpu.compiled_tiers[{index}].kernel_families: {name} " + f"must equal {sorted(expected_kernel_families)!r}" + ) + bits = set(tier.get("required_cpu_bits", [])) + expected_bits = set(policy["bits"][name]) + if bits != expected_bits: + errors.append( + f"$.cpu.compiled_tiers[{index}].required_cpu_bits: {name} " + f"must equal {sorted(expected_bits)!r}" + ) + state_policy = policy["os_state"][name] + if isinstance(state_policy, dict): + required_state = set(state_policy.get(host.get("os"), set())) + else: + required_state = set(state_policy) + actual_state = set(tier.get("required_os_state", [])) + if actual_state != required_state: + errors.append( + f"$.cpu.compiled_tiers[{index}].required_os_state: OS-state " + f"probes for {name} " + f"must equal {sorted(required_state)!r}" + ) + return errors + + +def _cuda_policy(manifest: dict[str, Any]) -> list[str]: + backend = manifest.get("backend", {}) + cuda = manifest.get("cuda") + backend_name = backend.get("name") if isinstance(backend, dict) else None + if backend_name != "cuda": + return ["$.cuda: only CUDA artifacts may carry SM evidence"] if cuda is not None else [] + if not isinstance(cuda, dict): + return ["$.cuda: CUDA artifact requires compiled SM and per-SM evidence"] + compiled = cuda.get("compiled_sms", []) + rows = cuda.get("sm_evidence", []) + if not isinstance(compiled, list) or not isinstance(rows, list): + return [] + errors: list[str] = [] + unsupported = [sm for sm in compiled if sm not in PRIMARY_CUDA_SMS] + if unsupported: + errors.append(f"$.cuda.compiled_sms: unsupported CUDA SM claim {unsupported!r}") + artifact = manifest.get("artifact", {}) + kind = artifact.get("kind") if isinstance(artifact, dict) else None + if kind == "primary" and compiled != list(PRIMARY_CUDA_SMS): + errors.append("$.cuda.compiled_sms: primary CUDA artifact requires all ten supported SMs") + if kind == "diagnostic" and len(compiled) != 1: + errors.append("$.cuda.compiled_sms: diagnostic CUDA artifact requires exactly one SM") + row_sms = [row.get("sm") for row in rows if isinstance(row, dict)] + if row_sms != compiled: + errors.append("$.cuda.sm_evidence: rows must match compiled SMs exactly and in order") + for index, row in enumerate(rows): + if not isinstance(row, dict) or row.get("sm") not in AOT_AVAILABILITY: + continue + sm = row["sm"] + available = AOT_AVAILABILITY[sm] + if row.get("aot_available") is not available: + errors.append(f"$.cuda.sm_evidence[{index}]: AOT availability for sm_{sm} is incorrect") + if row.get("portable_fallback") is not (not available): + errors.append(f"$.cuda.sm_evidence[{index}]: portable fallback for sm_{sm} is incorrect") + aot = row.get("aot_evidence", {}) + if isinstance(aot, dict): + state = aot.get("state") + if not available and state != "not-applicable": + errors.append(f"$.cuda.sm_evidence[{index}]: unavailable AOT must be not-applicable") + if available and state == "not-applicable": + errors.append(f"$.cuda.sm_evidence[{index}]: available AOT cannot be not-applicable") + return errors + + +def _publication_policy(manifest: dict[str, Any]) -> list[str]: + artifact = manifest.get("artifact", {}) + build = manifest.get("build", {}) + evidence_set = manifest.get("evidence", {}) + supply_chain = manifest.get("supply_chain", {}) + if ( + not isinstance(artifact, dict) + or not isinstance(build, dict) + or not isinstance(evidence_set, dict) + or not isinstance(supply_chain, dict) + ): + return [] + channel = artifact.get("channel") + if channel not in {"preview", "experimental-preview", "stable"}: + return [] + errors = [] + for key in PUBLISHED_EVIDENCE: + evidence = evidence_set.get(key, {}) + if not isinstance(evidence, dict) or evidence.get("state") != "passed": + errors.append( + f"$.artifact.channel: {channel} publication requires passed {key} evidence" + ) + if build.get("source_clean") is not True: + errors.append(f"$.artifact.channel: {channel} publication requires a clean source tree") + if channel != "stable": + return errors + for key in STABLE_EVIDENCE: + evidence = evidence_set.get(key, {}) + if not isinstance(evidence, dict) or evidence.get("state") != "passed": + errors.append(f"$.artifact.channel: stable requires passed {key} evidence") + for key in STABLE_SUPPLY_CHAIN_EVIDENCE: + evidence = supply_chain.get(key, {}) + if not isinstance(evidence, dict) or evidence.get("state") != "passed": + errors.append( + "$.artifact.channel: stable supply-chain requires passed " + f"{key} evidence" + ) + cuda = manifest.get("cuda") + if isinstance(cuda, dict): + for index, row in enumerate(cuda.get("sm_evidence", [])): + if not isinstance(row, dict): + continue + if row.get("runtime_evidence", {}).get("state") != "passed": + errors.append(f"$.artifact.channel: stable CUDA requires passed runtime evidence for row {index}") + if row.get("aot_available") and row.get("aot_evidence", {}).get("state") != "passed": + errors.append(f"$.artifact.channel: stable CUDA requires passed AOT evidence for row {index}") + return errors + + +def validate_manifest( + manifest: dict[str, Any], schema: dict[str, Any], repo_root: Path +) -> list[str]: + errors = _schema_errors(manifest, schema, schema) + for path, value in _evidence_objects(manifest): + errors.extend(_validate_evidence(path, value)) + errors.extend(_artifact_policy(manifest)) + errors.extend(_backend_policy(manifest, repo_root)) + errors.extend(_dependency_policy(manifest)) + errors.extend(_build_policy(manifest)) + errors.extend(_cpu_policy(manifest)) + errors.extend(_cuda_policy(manifest)) + errors.extend(_publication_policy(manifest)) + return errors + + +def generate_manifest( + facts: dict[str, Any], repo_root: Path, schema: dict[str, Any] +) -> dict[str, Any]: + if not isinstance(facts, dict): + raise ManifestError("generation input must be a JSON object") + manifest = copy.deepcopy(facts) + manifest["schema"] = SCHEMA_ID + manifest["schema_version"] = SCHEMA_VERSION + backend = manifest.get("backend") + if not isinstance(backend, dict): + raise ManifestError("generation input requires backend object") + if "resolved_features" in backend: + raise ManifestError("resolved_features is generated, not accepted as input") + if backend.get("name") == "cuda": + cuda = manifest.get("cuda") + if not isinstance(cuda, dict) or not isinstance(cuda.get("compiled_sms"), list): + raise ManifestError("CUDA generation input requires cuda.compiled_sms") + backend["resolved_features"] = resolve_cuda_features(cuda["compiled_sms"], repo_root) + else: + backend["resolved_features"] = [] + errors = validate_manifest(manifest, schema, repo_root) + if errors: + raise ManifestError("\n".join(errors)) + return manifest + + +def _read_json(path: Path) -> Any: + with path.open(encoding="utf-8") as handle: + return json.load(handle, object_pairs_hook=_reject_duplicate_keys) + + +def _write_text(path: Path, text: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--schema", type=Path, default=DEFAULT_SCHEMA) + subparsers = parser.add_subparsers(dest="command", required=True) + generate = subparsers.add_parser("generate", help="validate build facts and write canonical manifest JSON") + generate.add_argument("--input", type=Path, required=True) + generate.add_argument("--output", type=Path, required=True) + validate = subparsers.add_parser("validate", help="validate an existing release manifest") + validate.add_argument("manifest", type=Path) + args = parser.parse_args(argv) + repo_root = Path(__file__).resolve().parents[1] + schema_path = args.schema if args.schema.is_absolute() else repo_root / args.schema + try: + schema = load_schema(schema_path) + if args.command == "generate": + manifest = generate_manifest(_read_json(args.input), repo_root, schema) + _write_text(args.output, canonical_json(manifest)) + print(f"wrote {args.output}") + else: + manifest = _read_json(args.manifest) + errors = validate_manifest(manifest, schema, repo_root) + if errors: + raise ManifestError("\n".join(errors)) + print(f"valid: {args.manifest}") + except (OSError, json.JSONDecodeError, KeyError, ManifestError) as error: + print(f"release manifest error: {error}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/scripts/fixtures/release_manifest/v1/cpu-input.json b/tests/scripts/fixtures/release_manifest/v1/cpu-input.json new file mode 100644 index 00000000..4ff652d0 --- /dev/null +++ b/tests/scripts/fixtures/release_manifest/v1/cpu-input.json @@ -0,0 +1,235 @@ +{ + "artifact": { + "channel": "preview", + "id": "linux-x86_64-glibc-cpu", + "kind": "primary", + "static_boundary": "static-core", + "version": "0.1.0-test" + }, + "backend": { + "flags": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": false, + "VLLM_CPP_CUDA_ARCHITECTURES": [], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": false, + "VLLM_CPP_VULKAN": false + }, + "gpu_driver_boundary": "not-applicable", + "name": "cpu" + }, + "build": { + "compiler": "GNU C++ 13.2.0", + "resolved_cmake_options": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": false, + "VLLM_CPP_CUDA_ARCHITECTURES": [], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": false, + "VLLM_CPP_VULKAN": false + }, + "source_clean": true, + "source_commit": "0123456789abcdef0123456789abcdef01234567", + "test_commands": [ + "cmake --build build --target vllm", + "ctest --test-dir build" + ], + "toolchain": "cmake-3.30+ninja-1.12" + }, + "cpu": { + "baseline": "portable-sse2", + "compiled_tiers": [ + { + "execution_evidence": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "kernel_families": [ + "matmul-elem-f32-bf16" + ], + "name": "portable-sse2", + "required_cpu_bits": [ + "sse2" + ], + "required_os_state": [] + }, + { + "execution_evidence": { + "command": "lddtree bin/vllm-server", + "reason": "synthetic dependency audit failure", + "result": "exit 1: undeclared libsynthetic.so", + "state": "failed", + "url": "https://example.invalid/evidence/failed" + }, + "kernel_families": [ + "matmul-elem-f16" + ], + "name": "sse2-f16c", + "required_cpu_bits": [ + "avx", + "f16c", + "osxsave", + "sse2" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm" + ] + }, + { + "execution_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "kernel_families": [ + "matmul-elem-f32-bf16-f16" + ], + "name": "avx2-f16c", + "required_cpu_bits": [ + "avx", + "avx2", + "f16c", + "osxsave" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm" + ] + }, + { + "execution_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "kernel_families": [ + "matmul-elem-f32-bf16" + ], + "name": "avx512f", + "required_cpu_bits": [ + "avx", + "avx512f", + "osxsave" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm", + "xcr0:opmask", + "xcr0:zmm_hi256", + "xcr0:hi16_zmm" + ] + } + ], + "selected_tier": "portable-sse2" + }, + "dependencies": [ + { + "bundled": false, + "kind": "library", + "linkage": "dynamic", + "name": "glibc", + "role": "runtime", + "version": "2.31" + } + ], + "evidence": { + "archive_smoke": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "build": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "correctness": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "dependency_audit": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "performance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "runtime": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + } + }, + "host": { + "abi": "glibc", + "abi_version": "2.31", + "arch": "x86_64", + "os": "linux" + }, + "supply_chain": { + "archive_checksum": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "licenses": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "provenance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sbom": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + } +} diff --git a/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json b/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json new file mode 100644 index 00000000..afdc5b43 --- /dev/null +++ b/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json @@ -0,0 +1,238 @@ +{ + "artifact": { + "channel": "preview", + "id": "linux-x86_64-glibc-cpu", + "kind": "primary", + "static_boundary": "static-core", + "version": "0.1.0-test" + }, + "backend": { + "flags": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": false, + "VLLM_CPP_CUDA_ARCHITECTURES": [], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": false, + "VLLM_CPP_VULKAN": false + }, + "gpu_driver_boundary": "not-applicable", + "name": "cpu", + "resolved_features": [] + }, + "build": { + "compiler": "GNU C++ 13.2.0", + "resolved_cmake_options": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": false, + "VLLM_CPP_CUDA_ARCHITECTURES": [], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": false, + "VLLM_CPP_VULKAN": false + }, + "source_clean": true, + "source_commit": "0123456789abcdef0123456789abcdef01234567", + "test_commands": [ + "cmake --build build --target vllm", + "ctest --test-dir build" + ], + "toolchain": "cmake-3.30+ninja-1.12" + }, + "cpu": { + "baseline": "portable-sse2", + "compiled_tiers": [ + { + "execution_evidence": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "kernel_families": [ + "matmul-elem-f32-bf16" + ], + "name": "portable-sse2", + "required_cpu_bits": [ + "sse2" + ], + "required_os_state": [] + }, + { + "execution_evidence": { + "command": "lddtree bin/vllm-server", + "reason": "synthetic dependency audit failure", + "result": "exit 1: undeclared libsynthetic.so", + "state": "failed", + "url": "https://example.invalid/evidence/failed" + }, + "kernel_families": [ + "matmul-elem-f16" + ], + "name": "sse2-f16c", + "required_cpu_bits": [ + "avx", + "f16c", + "osxsave", + "sse2" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm" + ] + }, + { + "execution_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "kernel_families": [ + "matmul-elem-f32-bf16-f16" + ], + "name": "avx2-f16c", + "required_cpu_bits": [ + "avx", + "avx2", + "f16c", + "osxsave" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm" + ] + }, + { + "execution_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "kernel_families": [ + "matmul-elem-f32-bf16" + ], + "name": "avx512f", + "required_cpu_bits": [ + "avx", + "avx512f", + "osxsave" + ], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm", + "xcr0:opmask", + "xcr0:zmm_hi256", + "xcr0:hi16_zmm" + ] + } + ], + "selected_tier": "portable-sse2" + }, + "dependencies": [ + { + "bundled": false, + "kind": "library", + "linkage": "dynamic", + "name": "glibc", + "role": "runtime", + "version": "2.31" + } + ], + "evidence": { + "archive_smoke": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "build": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "correctness": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "dependency_audit": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "performance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "runtime": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + } + }, + "host": { + "abi": "glibc", + "abi_version": "2.31", + "arch": "x86_64", + "os": "linux" + }, + "schema": "vllm.cpp.release-manifest.v1", + "schema_version": 1, + "supply_chain": { + "archive_checksum": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "licenses": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "provenance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sbom": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + } +} diff --git a/tests/scripts/fixtures/release_manifest/v1/cuda-input.json b/tests/scripts/fixtures/release_manifest/v1/cuda-input.json new file mode 100644 index 00000000..613a6a1a --- /dev/null +++ b/tests/scripts/fixtures/release_manifest/v1/cuda-input.json @@ -0,0 +1,377 @@ +{ + "artifact": { + "channel": "preview", + "id": "linux-x86_64-glibc-cuda-fat", + "kind": "primary", + "static_boundary": "static-core", + "version": "0.1.0-test" + }, + "backend": { + "flags": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": true, + "VLLM_CPP_CUDA_ARCHITECTURES": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": true, + "VLLM_CPP_VULKAN": false + }, + "gpu_driver_boundary": "external-host-never-bundled", + "name": "cuda" + }, + "build": { + "compiler": "GNU C++ 13.2.0", + "resolved_cmake_options": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": true, + "VLLM_CPP_CUDA_ARCHITECTURES": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": true, + "VLLM_CPP_VULKAN": false + }, + "source_clean": true, + "source_commit": "0123456789abcdef0123456789abcdef01234567", + "test_commands": [ + "cmake --build build --target vllm", + "ctest --test-dir build" + ], + "toolchain": "cmake-3.30+ninja-1.12" + }, + "cuda": { + "compiled_sms": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "sm_evidence": [ + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "80" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "86" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "87" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "89" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "90a" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "100a" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "103a" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "110" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "120a" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "121a" + } + ] + }, + "dependencies": [ + { + "bundled": false, + "kind": "driver", + "linkage": "external", + "name": "nvidia-driver", + "role": "external-runtime", + "version": ">=580" + }, + { + "bundled": false, + "kind": "library", + "linkage": "dynamic", + "name": "glibc", + "role": "runtime", + "version": "2.31" + } + ], + "evidence": { + "archive_smoke": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "build": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "correctness": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "dependency_audit": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "performance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "runtime": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + }, + "host": { + "abi": "glibc", + "abi_version": "2.31", + "arch": "x86_64", + "os": "linux" + }, + "supply_chain": { + "archive_checksum": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "licenses": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "provenance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sbom": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + } +} diff --git a/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json b/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json new file mode 100644 index 00000000..07594ade --- /dev/null +++ b/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json @@ -0,0 +1,438 @@ +{ + "artifact": { + "channel": "preview", + "id": "linux-x86_64-glibc-cuda-fat", + "kind": "primary", + "static_boundary": "static-core", + "version": "0.1.0-test" + }, + "backend": { + "flags": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": true, + "VLLM_CPP_CUDA_ARCHITECTURES": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": true, + "VLLM_CPP_VULKAN": false + }, + "gpu_driver_boundary": "external-host-never-bundled", + "name": "cuda", + "resolved_features": [ + { + "compiled_sms": [ + "120a", + "121a" + ], + "name": "fp4-mma" + }, + { + "compiled_sms": [ + "120a", + "121a" + ], + "name": "cutlass-nvfp4" + }, + { + "compiled_sms": [ + "100a" + ], + "name": "cutlass-nvfp4-sm100" + }, + { + "compiled_sms": [ + "120a", + "121a" + ], + "name": "cutlass-fp8" + }, + { + "compiled_sms": [ + "90a" + ], + "name": "scaledmm-c3x-sm90" + }, + { + "compiled_sms": [ + "100a" + ], + "name": "scaledmm-c3x-sm100" + }, + { + "compiled_sms": [ + "120a", + "121a" + ], + "name": "marlin-nvfp4" + }, + { + "compiled_sms": [ + "80", + "86", + "87", + "89", + "120a", + "121a" + ], + "name": "fa2" + } + ] + }, + "build": { + "compiler": "GNU C++ 13.2.0", + "resolved_cmake_options": { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": true, + "VLLM_CPP_BUILD_TESTS": true, + "VLLM_CPP_CUDA": true, + "VLLM_CPP_CUDA_ARCHITECTURES": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "VLLM_CPP_HIP": false, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": false, + "VLLM_CPP_MLX": false, + "VLLM_CPP_SERVER": true, + "VLLM_CPP_TRITON": true, + "VLLM_CPP_VULKAN": false + }, + "source_clean": true, + "source_commit": "0123456789abcdef0123456789abcdef01234567", + "test_commands": [ + "cmake --build build --target vllm", + "ctest --test-dir build" + ], + "toolchain": "cmake-3.30+ninja-1.12" + }, + "cuda": { + "compiled_sms": [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a" + ], + "sm_evidence": [ + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "80" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "86" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "87" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "89" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "90a" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "100a" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "103a" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "110" + }, + { + "aot_available": false, + "aot_evidence": { + "command": "", + "reason": "synthetic cross-build has no matching runtime host", + "result": "", + "state": "not-applicable", + "url": "" + }, + "portable_fallback": true, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "120a" + }, + { + "aot_available": true, + "aot_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "portable_fallback": false, + "runtime_evidence": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sm": "121a" + } + ] + }, + "dependencies": [ + { + "bundled": false, + "kind": "driver", + "linkage": "external", + "name": "nvidia-driver", + "role": "external-runtime", + "version": ">=580" + }, + { + "bundled": false, + "kind": "library", + "linkage": "dynamic", + "name": "glibc", + "role": "runtime", + "version": "2.31" + } + ], + "evidence": { + "archive_smoke": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "build": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "correctness": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "dependency_audit": { + "command": "cmake --build build --target vllm", + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://example.invalid/evidence/passed" + }, + "performance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "runtime": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + }, + "host": { + "abi": "glibc", + "abi_version": "2.31", + "arch": "x86_64", + "os": "linux" + }, + "schema": "vllm.cpp.release-manifest.v1", + "schema_version": 1, + "supply_chain": { + "archive_checksum": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "licenses": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "provenance": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + }, + "sbom": { + "command": "", + "reason": "synthetic fixture has not run this gate", + "result": "", + "state": "absent", + "url": "" + } + } +} diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 21dc4f69..89126bd5 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -6,9 +6,11 @@ import importlib.util import re import sys +import tempfile import unittest from dataclasses import replace from pathlib import Path +from unittest import mock ROOT = Path(__file__).resolve().parents[2] @@ -218,6 +220,35 @@ def test_tables_require_semantic_owner_column(self) -> None: r"MODEL-FACTORY-registry table lacks semantic owner column", ) + def test_engine_summary_rejects_stale_area_rollup(self) -> None: + source = agent_record.ENGINE_MATRIX.read_text(encoding="utf-8") + current = next( + line + for line in source.splitlines() + if line.startswith("| Serving, API, CLI, library |") + ) + cells = [cell.strip() for cell in current.strip().strip("|").split("|")] + self.assertGreater(int(cells[6]), 0) + cells[5] = str(int(cells[5]) + 1) + cells[6] = str(int(cells[6]) - 1) + stale = "| " + " | ".join(cells) + " |" + + with tempfile.TemporaryDirectory(dir=ROOT) as temp_dir: + matrix = Path(temp_dir) / "engine-matrix.md" + with mock.patch.object(agent_record, "ENGINE_MATRIX", matrix): + matrix.write_text(source, encoding="utf-8") + baseline_errors: list[str] = [] + baseline_rows = agent_record.parse_claim_rows(matrix, baseline_errors) + agent_record.check_engine_summary(baseline_rows, baseline_errors) + self.assertEqual(baseline_errors, []) + + matrix.write_text(source.replace(current, stale), encoding="utf-8") + errors: list[str] = [] + rows = agent_record.parse_claim_rows(matrix, errors) + agent_record.check_engine_summary(rows, errors) + + require(errors, r"Serving, API, CLI, library summary ready=\d+; actual \d+") + if __name__ == "__main__": unittest.main() diff --git a/tests/scripts/test_check_pr_size.py b/tests/scripts/test_check_pr_size.py index 2830277b..bdf4a4b0 100755 --- a/tests/scripts/test_check_pr_size.py +++ b/tests/scripts/test_check_pr_size.py @@ -24,12 +24,18 @@ class PathClassification(unittest.TestCase): def test_each_mutable_surface_has_an_explicit_class(self) -> None: expected = { "src/vt/x.cpp": "product", + "scripts/check-release-binary-contract.py": "product", "scripts/check-policy.py": "governance_checker", "tests/scripts/test_policy_contract.py": "governance_test", ".agents/policy.csv": "policy", ".agents/state.md": "append_only_record", "docs/STATUS.md": "public_document", + "MANIFESTO.md": "public_document", + "CLAUDE.md": "procedure", ".github/workflows/ci.yml": "ci", + "release/manifest-v1.schema.json": "configuration", + "scripts/env-doc-allowlist.txt": "configuration", + "tests/scripts/fixtures/release_manifest/v1/cpu-input.json": "asset", } for path, path_class in expected.items(): with self.subTest(path=path): @@ -74,6 +80,10 @@ def test_similar_names_do_not_enter_governance_classes(self) -> None: checker.recognized_evidence("scripts/check-policy.py"), "tests/scripts/test_policy_contract_extra.py", ) + self.assertEqual( + checker.recognized_evidence("scripts/check-agent-record.py"), + "tests/scripts/test_agent_record.py", + ) class BudgetEnforcement(unittest.TestCase): @@ -317,6 +327,7 @@ def test_production_pr_classifier_covers_every_governed_path_class(self) -> None ".github/workflows/ci.yml", "scripts/agent-role.py", ".env.example", + "release/manifest-v1.schema.json", ) for path in governed: with self.subTest(path=path): diff --git a/tests/scripts/test_check_public_doc_tables.py b/tests/scripts/test_check_public_doc_tables.py index 37b3d4d1..df5e2aea 100644 --- a/tests/scripts/test_check_public_doc_tables.py +++ b/tests/scripts/test_check_public_doc_tables.py @@ -431,3 +431,9 @@ def test_the_live_page_is_inside_its_ratchet(self) -> None: self.assertEqual( doc_tables.status_errors(doc_tables.STATUS.read_text(encoding="utf-8")), [] ) + + def test_the_live_page_keeps_the_character_ratchet_tight(self) -> None: + text = doc_tables.STATUS.read_text(encoding="utf-8") + slack = doc_tables.STATUS_RATCHET["chars"] - len(text) + self.assertGreaterEqual(slack, 0) + self.assertLessEqual(slack, 25) diff --git a/tests/scripts/test_check_release_binary_contract.py b/tests/scripts/test_check_release_binary_contract.py index 5cdde14d..99749e78 100644 --- a/tests/scripts/test_check_release_binary_contract.py +++ b/tests/scripts/test_check_release_binary_contract.py @@ -127,75 +127,78 @@ RECORD_ANCHORS = { ".agents/engine-matrix.md": "| `ENG-RELEASE-BINARIES` |", ".agents/roadmap_v1.md": "| REL | `ROAD-V1-RELEASE` |", - ".agents/NOW.md": "| Release | SPIKE; 30/30 | #129 |", + ".agents/NOW.md": ( + "| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** |" + ), ".agents/coordination.md": ( - "| `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | " - "`ENG-RELEASE-BINARIES` |" + "| `CLAIM-ENG-RELEASE-BINARIES-W5` | " + "`ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) |" ), ".agents/state.md": ( - "## 2026-08-07 — Release matrix revised: fat CUDA and adaptive CPU " - "are the primary downloads" - ), - "docs/STATUS.md": ( - "#129: SPIKE∅" + "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE" ), + "docs/STATUS.md": "#141 W5 19/19 ACTIVE ART∅", "docs/BENCHMARKS.md": ( - "| **Binary release matrix (spiked)** | `ENG-RELEASE-BINARIES`:" + "| **Binary release matrix (ACTIVE; W5 implemented)** | " + "`ENG-RELEASE-BINARIES`:" ), } LIFECYCLE_RECORD_MUTATIONS = ( ( ".agents/engine-matrix.md", - "`SPIKE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` |", - "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` |", + "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", + "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", "engine-matrix release lifecycle", ), ( ".agents/engine-matrix.md", - "gaps remain; no install/archive/publish implementation", - "gaps closed; install/archive/publish implementation complete", + "install/archive/publish implementation and all real release evidence " + "remain pending", + "install/archive/publish implementation and all real release evidence " + "are complete", "engine-matrix release lifecycle", ), ( ".agents/roadmap_v1.md", - "`SPIKE` | Fresh review of PR #129", - "`DONE` | Fresh review of PR #129", + "`ACTIVE` | W5 versioned manifest schema/generator", + "`DONE` | W5 versioned manifest schema/generator", "roadmap release lifecycle", ), ( ".agents/roadmap_v1.md", - "bundle work; no archive exists", - "bundle work complete; archive exists", + "no archive exists", + "archive exists", "roadmap release lifecycle", ), ( ".agents/coordination.md", - "| `ACTIVE` | 2026-08-07 — user-reviewed revision complete: primary fat " - "CUDA + adaptive CPU per host ABI, optional per-SM diagnostics; row stays " - "`SPIKE`; awaiting fresh review |", - "| `DONE` | 2026-08-07 — user-reviewed revision complete: primary fat " - "CUDA + adaptive CPU per host ABI, optional per-SM diagnostics; row stays " - "`SPIKE`; awaiting fresh review |", + "| `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals " + "10/10 killed; accepted release suite 30/30; no archive or real " + "runtime/correctness/performance evidence |", + "| `DONE` | 2026-08-08 — W5 19/19; fresh-review production removals " + "10/10 killed; accepted release suite 30/30; " + "archive and runtime/correctness/performance evidence complete |", "coordination release lifecycle", ), ( ".agents/coordination.md", - "no CMake, workflow, source, test, or artifact implementation", - "CMake, workflow, source, test, and artifact implementation complete", + "Excludes W1-W4, W6-W13, archives, install/package/publish workflows and " + "runtime artifacts", + "Includes W1-W13, archives, install/package/publish workflows and " + "runtime artifacts", "coordination release lifecycle", ), ( ".agents/coordination.md", - "row stays `SPIKE`; awaiting fresh review", - "row is `DONE`; release shipped", + "no archive or real runtime/correctness/performance evidence", + "release archive and runtime/correctness/performance evidence shipped", "coordination release lifecycle", ), ( ".agents/state.md", - "`ENG-RELEASE-BINARIES` remains `SPIKE`, and no archive or implementation " - "is\nclaimed.", - "`ENG-RELEASE-BINARIES` is `DONE`, with archive and implementation.", + "`ACTIVE`, never `DONE` or `GATING`.", + "`DONE`, with every release gate complete.", "state release lifecycle", ), ) @@ -235,17 +238,16 @@ PUBLIC_PENDING_MUTATIONS = ( ( "docs/BENCHMARKS.md", - "**PENDING:** pins 10-SM fat CUDA, adaptive no-AVX2 CPU, " - "W1-W13/W10-W12 policy, public pending states; 30 tests GREEN. No archive, " - "staged smoke, runtime, correctness, or performance evidence", + "**PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/" + "correctness/performance gates", "**SHIPPED:** archive, runtime, correctness, and performance evidence " "complete", "docs/BENCHMARKS.md release row", ), ( "docs/STATUS.md", - "Supported (subset); #129: SPIKE∅", - "Supported; #129: SHIPPED", + "Subset; #141 W5 19/19 ACTIVE ART∅", + "Supported; #141: RELEASE DONE/ARTIFACTS✓", "docs/STATUS.md release row", ), ) @@ -283,7 +285,9 @@ ) EXACT_MACHINE_FIELDS = { - "lifecycle": "SPIKE", + "lifecycle": "ACTIVE", + "manifest_schema": "vllm.cpp.release-manifest.v1", + "work_W5_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -297,7 +301,8 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/state.md,docs/STATUS.md," - "docs/BENCHMARKS.md" + "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" ), } @@ -812,8 +817,8 @@ def test_unknown_machine_fields_are_fail_closed(self) -> None: mutate( root, self.SPEC, - "lifecycle=SPIKE", - f"lifecycle=SPIKE\n{field}={value}", + "lifecycle=ACTIVE", + f"lifecycle=ACTIVE\n{field}={value}", ) bypass_unknown_field_enforcement(root) result = run_checker(root) diff --git a/tests/scripts/test_release_manifest.py b/tests/scripts/test_release_manifest.py new file mode 100644 index 00000000..1e317b64 --- /dev/null +++ b/tests/scripts/test_release_manifest.py @@ -0,0 +1,898 @@ +#!/usr/bin/env python3 +"""Executable W5 contract for the versioned binary-release manifest.""" + +from __future__ import annotations + +import copy +import importlib.util +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts" / "release_manifest.py" +SCHEMA = ROOT / "release" / "manifest-v1.schema.json" +FIXTURES = ROOT / "tests" / "scripts" / "fixtures" / "release_manifest" / "v1" +PRIMARY_SMS = ["80", "86", "87", "89", "90a", "100a", "103a", "110", "120a", "121a"] +AOT_SMS = {"80", "86", "89", "90a", "100a", "121a"} +EVIDENCE_KEYS = ( + "build", + "archive_smoke", + "dependency_audit", + "runtime", + "correctness", + "performance", +) + + +def evidence(state: str) -> dict[str, str]: + values = { + "absent": { + "reason": "synthetic fixture has not run this gate", + "command": "", + "result": "", + "url": "", + }, + "not-applicable": { + "reason": "synthetic cross-build has no matching runtime host", + "command": "", + "result": "", + "url": "", + }, + "failed": { + "reason": "synthetic dependency audit failure", + "command": "lddtree bin/vllm-server", + "result": "exit 1: undeclared libsynthetic.so", + "url": "https://example.invalid/evidence/failed", + }, + "passed": { + "reason": "", + "command": "cmake --build build --target vllm", + "result": "exit 0", + "url": "https://example.invalid/evidence/passed", + }, + } + return {"state": state, **values[state]} + + +def flags(backend: str, sms: list[str] | None = None) -> dict[str, object]: + return { + "MLX_ROOT": "", + "VLLM_CPP_BUILD_EXAMPLES": True, + "VLLM_CPP_BUILD_TESTS": True, + "VLLM_CPP_CUDA": backend == "cuda", + "VLLM_CPP_CUDA_ARCHITECTURES": list(sms or []), + "VLLM_CPP_HIP": False, + "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_METAL": backend in {"metal", "mlx"}, + "VLLM_CPP_MLX": backend == "mlx", + "VLLM_CPP_SERVER": True, + "VLLM_CPP_TRITON": backend == "cuda", + "VLLM_CPP_VULKAN": backend == "vulkan", + } + + +def build_metadata(resolved_flags: dict[str, object]) -> dict[str, object]: + return { + "source_commit": "0123456789abcdef0123456789abcdef01234567", + "source_clean": True, + "compiler": "GNU C++ 13.2.0", + "toolchain": "cmake-3.30+ninja-1.12", + "resolved_cmake_options": copy.deepcopy(resolved_flags), + "test_commands": ["cmake --build build --target vllm", "ctest --test-dir build"], + } + + +def supply_chain() -> dict[str, object]: + return { + "archive_checksum": evidence("absent"), + "sbom": evidence("absent"), + "provenance": evidence("absent"), + "licenses": evidence("absent"), + } + + +def cpu_facts() -> dict[str, object]: + resolved_flags = flags("cpu") + tiers = [ + { + "name": "portable-sse2", + "kernel_families": ["matmul-elem-f32-bf16"], + "required_cpu_bits": ["sse2"], + "required_os_state": [], + "execution_evidence": evidence("passed"), + }, + { + "name": "sse2-f16c", + "kernel_families": ["matmul-elem-f16"], + "required_cpu_bits": ["avx", "f16c", "osxsave", "sse2"], + "required_os_state": ["xcr0:xmm", "xcr0:ymm"], + "execution_evidence": evidence("failed"), + }, + { + "name": "avx2-f16c", + "kernel_families": ["matmul-elem-f32-bf16-f16"], + "required_cpu_bits": ["avx", "avx2", "f16c", "osxsave"], + "required_os_state": ["xcr0:xmm", "xcr0:ymm"], + "execution_evidence": evidence("absent"), + }, + { + "name": "avx512f", + "kernel_families": ["matmul-elem-f32-bf16"], + "required_cpu_bits": ["avx", "avx512f", "osxsave"], + "required_os_state": [ + "xcr0:xmm", + "xcr0:ymm", + "xcr0:opmask", + "xcr0:zmm_hi256", + "xcr0:hi16_zmm", + ], + "execution_evidence": evidence("not-applicable"), + }, + ] + return { + "artifact": { + "id": "linux-x86_64-glibc-cpu", + "version": "0.1.0-test", + "channel": "preview", + "kind": "primary", + "static_boundary": "static-core", + }, + "host": { + "os": "linux", + "arch": "x86_64", + "abi": "glibc", + "abi_version": "2.31", + }, + "backend": { + "name": "cpu", + "flags": resolved_flags, + "gpu_driver_boundary": "not-applicable", + }, + "dependencies": [ + { + "name": "glibc", + "version": "2.31", + "kind": "library", + "linkage": "dynamic", + "bundled": False, + "role": "runtime", + } + ], + "cpu": { + "baseline": "portable-sse2", + "selected_tier": "portable-sse2", + "compiled_tiers": tiers, + }, + "build": build_metadata(resolved_flags), + "supply_chain": supply_chain(), + "evidence": { + "build": evidence("passed"), + "archive_smoke": evidence("passed"), + "dependency_audit": evidence("passed"), + "runtime": evidence("not-applicable"), + "correctness": evidence("absent"), + "performance": evidence("absent"), + }, + } + + +def cuda_facts() -> dict[str, object]: + resolved_flags = flags("cuda", PRIMARY_SMS) + sm_evidence = [] + for sm in PRIMARY_SMS: + available = sm in AOT_SMS + sm_evidence.append( + { + "sm": sm, + "aot_available": available, + "portable_fallback": not available, + "aot_evidence": evidence("absent" if available else "not-applicable"), + "runtime_evidence": evidence("absent"), + } + ) + return { + "artifact": { + "id": "linux-x86_64-glibc-cuda-fat", + "version": "0.1.0-test", + "channel": "preview", + "kind": "primary", + "static_boundary": "static-core", + }, + "host": { + "os": "linux", + "arch": "x86_64", + "abi": "glibc", + "abi_version": "2.31", + }, + "backend": { + "name": "cuda", + "flags": resolved_flags, + "gpu_driver_boundary": "external-host-never-bundled", + }, + "dependencies": [ + { + "name": "nvidia-driver", + "version": ">=580", + "kind": "driver", + "linkage": "external", + "bundled": False, + "role": "external-runtime", + }, + { + "name": "glibc", + "version": "2.31", + "kind": "library", + "linkage": "dynamic", + "bundled": False, + "role": "runtime", + }, + ], + "cuda": {"compiled_sms": list(PRIMARY_SMS), "sm_evidence": sm_evidence}, + "build": build_metadata(resolved_flags), + "supply_chain": supply_chain(), + "evidence": { + key: ( + evidence("passed") + if key in {"build", "archive_smoke", "dependency_audit"} + else evidence("absent") + ) + for key in EVIDENCE_KEYS + }, + } + + +def load_tool(): + spec = importlib.util.spec_from_file_location("release_manifest", TOOL) + if spec is None or spec.loader is None: + raise AssertionError(f"cannot import {TOOL}") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +class ReleaseManifestTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.tool = None + cls.schema = None + + def setUp(self) -> None: + self.assertTrue(SCHEMA.is_file(), f"missing versioned schema: {SCHEMA}") + self.assertTrue(TOOL.is_file(), f"missing build-time generator: {TOOL}") + if self.__class__.tool is None: + self.__class__.tool = load_tool() + self.__class__.schema = self.tool.load_schema(SCHEMA) + + def generated(self, facts: dict[str, object]) -> dict[str, object]: + manifest = self.tool.generate_manifest(copy.deepcopy(facts), ROOT, self.schema) + errors = self.tool.validate_manifest(manifest, self.schema, ROOT) + self.assertEqual(errors, []) + return manifest + + def assert_invalid(self, manifest: dict[str, object], needle: str = "") -> None: + errors = self.tool.validate_manifest(manifest, self.schema, ROOT) + self.assertTrue(errors, "mutation unexpectedly validated") + if needle: + self.assertTrue(any(needle in error for error in errors), errors) + + def test_cpu_and_cuda_generation_is_deterministic_and_matches_goldens(self) -> None: + for stem, facts in (("cpu", cpu_facts()), ("cuda", cuda_facts())): + manifest = self.generated(facts) + canonical = self.tool.canonical_json(manifest) + self.assertEqual(canonical, self.tool.canonical_json(manifest)) + self.assertEqual(canonical, (FIXTURES / f"{stem}-manifest.json").read_text()) + self.assertEqual(facts, json.loads((FIXTURES / f"{stem}-input.json").read_text())) + + def test_cli_generate_and_validate_use_only_stdlib_build_time_tooling(self) -> None: + with tempfile.TemporaryDirectory() as td: + output = Path(td) / "manifest.json" + generated = subprocess.run( + [sys.executable, str(TOOL), "generate", "--input", str(FIXTURES / "cpu-input.json"), "--output", str(output)], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(generated.returncode, 0, generated.stdout + generated.stderr) + self.assertEqual(output.read_text(), (FIXTURES / "cpu-manifest.json").read_text()) + validated = subprocess.run( + [sys.executable, str(TOOL), "validate", str(output)], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(validated.returncode, 0, validated.stdout + validated.stderr) + + def test_w5_suite_registration_is_fail_closed(self) -> None: + checker_path = ROOT / "scripts" / "check-release-binary-contract.py" + spec = importlib.util.spec_from_file_location("release_contract_checker_w5", checker_path) + self.assertIsNotNone(spec) + self.assertIsNotNone(spec.loader) + checker = importlib.util.module_from_spec(spec) + spec.loader.exec_module(checker) + preflight = (ROOT / "scripts" / "agent-preflight.sh").read_text() + ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text() + self.assertEqual(checker.wiring_errors(preflight, ci), []) + mutations = ( + (preflight.replace(" test_release_manifest\n", "", 1), ci, "preflight"), + (preflight, ci.replace(" python3 tests/scripts/test_release_manifest.py\n", "", 1), "CI"), + ) + for mutated_preflight, mutated_ci, reason in mutations: + with self.subTest(reason=reason): + errors = checker.wiring_errors(mutated_preflight, mutated_ci) + self.assertTrue(any("W5 manifest suite" in error for error in errors), errors) + + def test_schema_required_keys_enums_and_additional_properties_are_live(self) -> None: + base = self.generated(cpu_facts()) + required_paths = [ + ((), "schema"), ((), "schema_version"), ((), "artifact"), + ((), "host"), ((), "backend"), ((), "dependencies"), ((), "build"), + ((), "supply_chain"), ((), "evidence"), + (("artifact",), "id"), (("artifact",), "version"), + (("artifact",), "channel"), (("artifact",), "kind"), + (("artifact",), "static_boundary"), (("host",), "os"), (("host",), "arch"), + (("host",), "abi"), (("host",), "abi_version"), (("backend",), "name"), + (("backend",), "flags"), (("backend",), "resolved_features"), + (("backend",), "gpu_driver_boundary"), (("cpu",), "baseline"), + (("cpu",), "selected_tier"), (("cpu",), "compiled_tiers"), + (("build",), "source_commit"), (("build",), "source_clean"), + (("build",), "compiler"), (("build",), "toolchain"), + (("build",), "resolved_cmake_options"), (("build",), "test_commands"), + (("supply_chain",), "archive_checksum"), (("supply_chain",), "sbom"), + (("supply_chain",), "provenance"), (("supply_chain",), "licenses"), + (("dependencies", 0), "name"), + (("dependencies", 0), "version"), (("dependencies", 0), "kind"), + (("dependencies", 0), "linkage"), (("dependencies", 0), "bundled"), + (("dependencies", 0), "role"), + ] + for path, key in required_paths: + with self.subTest(path=path, key=key): + mutant = copy.deepcopy(base) + target = mutant + for part in path: + target = target[part] + del target[key] + self.assert_invalid(mutant, "required") + for path, value in ( + (("artifact", "channel"), "nightly"), + (("artifact", "kind"), "universal"), + (("artifact", "static_boundary"), "fully-static-gpu"), + (("host", "os"), "windows"), + (("host", "arch"), "riscv64"), + (("backend", "name"), "rocm"), + ): + with self.subTest(path=path): + mutant = copy.deepcopy(base) + mutant[path[0]][path[1]] = value + self.assert_invalid(mutant) + for path in ((), ("artifact",), ("host",), ("backend",), ("backend", "flags"), ("build",), ("supply_chain",), ("cpu",), ("cpu", "compiled_tiers", 0), ("dependencies", 0), ("evidence", "build")): + with self.subTest(unknown_at=path): + mutant = copy.deepcopy(base) + target = mutant + for part in path: + target = target[part] + target["unknown_w5_field"] = "must fail" + self.assert_invalid(mutant, "unknown") + + def test_schema_type_enum_and_const_semantics_are_independently_live(self) -> None: + self.assertTrue( + self.tool._schema_errors("not-an-integer", {"type": "integer"}, {}), + "schema type enforcement is not live", + ) + self.assertTrue( + self.tool._schema_errors("not-in-enum", {"enum": ["allowed"]}, {}), + "schema enum enforcement is not live", + ) + self.assertTrue( + self.tool._schema_errors(2, {"const": 1}, {}), + "schema const enforcement is not live", + ) + manifest = self.generated(cpu_facts()) + manifest["schema_version"] = True + self.assert_invalid(manifest, "schema_version") + + def test_boolean_is_neither_an_integer_type_nor_integer_json_constant(self) -> None: + self.assertFalse( + self.tool._type_matches(True, "integer"), + "JSON boolean must not satisfy the integer schema type", + ) + self.assertFalse( + self.tool._json_equal(True, 1), + "JSON boolean and integer constants must remain distinct", + ) + + def test_evidence_states_remain_four_distinct_fail_closed_values(self) -> None: + manifest = self.generated(cpu_facts()) + states = {manifest["evidence"][key]["state"] for key in EVIDENCE_KEYS} + states.update(tier["execution_evidence"]["state"] for tier in manifest["cpu"]["compiled_tiers"]) + self.assertEqual(states, {"absent", "not-applicable", "failed", "passed"}) + for key in EVIDENCE_KEYS: + for field in ("state", "reason", "command", "result", "url"): + with self.subTest(evidence=key, missing=field): + mutant = copy.deepcopy(manifest) + del mutant["evidence"][key][field] + self.assert_invalid(mutant) + semantic_mutations = ( + ("passed", "command", ""), ("passed", "result", ""), ("passed", "url", ""), + ("failed", "reason", ""), ("failed", "command", ""), ("failed", "result", ""), + ("absent", "reason", ""), ("not-applicable", "reason", ""), + ("absent", "command", "should-not-run"), + ("not-applicable", "result", "truthy-collapse"), + ) + for state, field, value in semantic_mutations: + with self.subTest(state=state, field=field): + mutant = copy.deepcopy(manifest) + mutant["evidence"]["build"] = evidence(state) + mutant["evidence"]["build"][field] = value + self.assert_invalid(mutant, state) + mutant = copy.deepcopy(manifest) + mutant["evidence"]["build"]["state"] = False + self.assert_invalid(mutant) + + def test_identity_build_supply_chain_and_duplicate_json_are_fail_closed(self) -> None: + manifest = self.generated(cpu_facts()) + for path, value in ( + (("artifact", "version"), ""), + (("build", "source_commit"), "not-a-commit"), + (("build", "compiler"), ""), + (("build", "toolchain"), ""), + (("build", "test_commands"), []), + ): + with self.subTest(path=path): + mutant = copy.deepcopy(manifest) + mutant[path[0]][path[1]] = value + self.assert_invalid(mutant) + mutant = copy.deepcopy(manifest) + mutant["build"]["resolved_cmake_options"]["VLLM_CPP_BUILD_TESTS"] = False + self.assert_invalid(mutant, "resolved CMake") + duplicate_dep = copy.deepcopy(manifest) + duplicate_dep["dependencies"].append(copy.deepcopy(duplicate_dep["dependencies"][0])) + self.assert_invalid(duplicate_dep, "unique") + duplicate_test = copy.deepcopy(manifest) + duplicate_test["build"]["test_commands"].append(duplicate_test["build"]["test_commands"][0]) + self.assert_invalid(duplicate_test, "unique") + with tempfile.TemporaryDirectory() as td: + duplicate_json = Path(td) / "duplicate.json" + text = self.tool.canonical_json(manifest) + duplicate_json.write_text(text.replace(' "schema": ', ' "schema": "duplicate",\n "schema": ', 1)) + result = subprocess.run( + [sys.executable, str(TOOL), "validate", str(duplicate_json)], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("duplicate JSON key", result.stderr) + + def test_generator_has_no_cwd_environment_locale_or_timestamp_leakage(self) -> None: + facts = cpu_facts() + before = self.tool.canonical_json(self.generated(facts)) + old_cwd = Path.cwd() + old_env = os.environ.get("W5_MANIFEST_LEAK_SENTINEL") + try: + with tempfile.TemporaryDirectory() as td: + os.chdir(td) + os.environ["W5_MANIFEST_LEAK_SENTINEL"] = "must-not-appear" + after = self.tool.canonical_json(self.generated(facts)) + finally: + os.chdir(old_cwd) + if old_env is None: + os.environ.pop("W5_MANIFEST_LEAK_SENTINEL", None) + else: + os.environ["W5_MANIFEST_LEAK_SENTINEL"] = old_env + self.assertEqual(after, before) + self.assertNotIn("must-not-appear", after) + self.assertNotIn(str(old_cwd), after) + self.assertNotIn("timestamp", after.lower()) + + def test_every_current_x86_tier_and_its_contract_are_mandatory(self) -> None: + manifest = self.generated(cpu_facts()) + tiers = manifest["cpu"]["compiled_tiers"] + for index, tier in enumerate(tiers): + with self.subTest(remove=tier["name"]): + mutant = copy.deepcopy(manifest) + del mutant["cpu"]["compiled_tiers"][index] + self.assert_invalid(mutant, "compiled tiers") + for field in ("name", "kernel_families", "required_cpu_bits", "required_os_state", "execution_evidence"): + with self.subTest(tier=tier["name"], missing=field): + mutant = copy.deepcopy(manifest) + del mutant["cpu"]["compiled_tiers"][index][field] + self.assert_invalid(mutant) + mutant = copy.deepcopy(manifest) + mutant["cpu"]["compiled_tiers"][1]["name"] = "amx-unavailable" + self.assert_invalid(mutant, "compiled tiers") + mutant = copy.deepcopy(manifest) + mutant["cpu"]["selected_tier"] = "amx-unavailable" + self.assert_invalid(mutant, "selected tier") + for field in ("kernel_families", "required_cpu_bits"): + mutant = copy.deepcopy(manifest) + mutant["cpu"]["compiled_tiers"][1][field] = [] + self.assert_invalid(mutant) + mutant = copy.deepcopy(manifest) + mutant["cpu"]["compiled_tiers"][1]["required_os_state"] = [] + self.assert_invalid(mutant, "OS-state") + + def test_cpu_tier_inventory_is_exact_on_x86_and_aarch64(self) -> None: + manifest = self.generated(cpu_facts()) + for index, tier in enumerate(manifest["cpu"]["compiled_tiers"]): + for field, extra in ( + ("kernel_families", "fabricated-kernel-family"), + ("required_cpu_bits", "fabricated-cpu-bit"), + ("required_os_state", "fabricated:os-probe"), + ): + with self.subTest(tier=tier["name"], extra=field): + mutant = copy.deepcopy(manifest) + mutant["cpu"]["compiled_tiers"][index][field].append(extra) + self.assertTrue( + any( + field in error + for error in self.tool._cpu_policy(mutant) + ), + self.tool._cpu_policy(mutant), + ) + for field in ( + "kernel_families", + "required_cpu_bits", + "required_os_state", + ): + if not tier[field]: + continue + with self.subTest(tier=tier["name"], missing=field): + mutant = copy.deepcopy(manifest) + del mutant["cpu"]["compiled_tiers"][index][field][0] + self.assertTrue( + any( + field in error + for error in self.tool._cpu_policy(mutant) + ), + self.tool._cpu_policy(mutant), + ) + + arm_kernel_families = { + "portable-neon": ["matmul-elem-f32-bf16-f16"], + "i8mm": ["quant-dot-q4_0-q8_0-q4_K-q6_K", "quant-repack-q8_0"], + } + for os_name, abi, artifact_id, probe in ( + ( + "linux", + "glibc", + "linux-aarch64-glibc-cpu", + "getauxval:AT_HWCAP2:HWCAP2_I8MM", + ), + ( + "macos", + "macos", + "macos-arm64-cpu-policy-probe", + "sysctl:hw.optional.arm.FEAT_I8MM", + ), + ): + arm = { + "artifact": {"id": artifact_id}, + "host": {"os": os_name, "arch": "aarch64", "abi": abi}, + "backend": {"name": "cpu"}, + "cpu": { + "baseline": "portable-neon", + "selected_tier": "portable-neon", + "compiled_tiers": [ + { + "name": "portable-neon", + "kernel_families": arm_kernel_families["portable-neon"], + "required_cpu_bits": ["neon"], + "required_os_state": [], + }, + { + "name": "i8mm", + "kernel_families": arm_kernel_families["i8mm"], + "required_cpu_bits": ["i8mm"], + "required_os_state": [probe], + }, + ], + }, + } + with self.subTest(aarch64_os=os_name): + self.assertEqual(self.tool._cpu_policy(arm), []) + wrong_probe = copy.deepcopy(arm) + wrong_probe["cpu"]["compiled_tiers"][1]["required_os_state"] = [ + "fabricated:os-probe" + ] + self.assertTrue( + any( + "required_os_state" in error + for error in self.tool._cpu_policy(wrong_probe) + ), + self.tool._cpu_policy(wrong_probe), + ) + + def test_primary_cuda_requires_every_sm_and_independent_aot_runtime_evidence(self) -> None: + manifest = self.generated(cuda_facts()) + self.assertEqual(manifest["cuda"]["compiled_sms"], PRIMARY_SMS) + for index, sm in enumerate(PRIMARY_SMS): + with self.subTest(remove_sm=sm): + mutant = copy.deepcopy(manifest) + del mutant["cuda"]["compiled_sms"][index] + del mutant["cuda"]["sm_evidence"][index] + mutant["backend"]["flags"]["VLLM_CPP_CUDA_ARCHITECTURES"] = mutant["cuda"]["compiled_sms"] + self.assert_invalid(mutant, "primary CUDA") + for field in ("sm", "aot_available", "portable_fallback", "aot_evidence", "runtime_evidence"): + with self.subTest(sm=sm, missing=field): + mutant = copy.deepcopy(manifest) + del mutant["cuda"]["sm_evidence"][index][field] + self.assert_invalid(mutant) + mutant = copy.deepcopy(manifest) + mutant["cuda"]["sm_evidence"][index]["aot_available"] = sm not in AOT_SMS + self.assert_invalid(mutant, "AOT") + independent = copy.deepcopy(manifest) + independent["cuda"]["sm_evidence"][0]["aot_evidence"] = evidence("passed") + independent["cuda"]["sm_evidence"][0]["runtime_evidence"] = evidence("absent") + self.assertEqual(self.tool.validate_manifest(independent, self.schema, ROOT), []) + mutant = copy.deepcopy(manifest) + mutant["cuda"]["compiled_sms"][0] = "75" + mutant["backend"]["flags"]["VLLM_CPP_CUDA_ARCHITECTURES"][0] = "75" + mutant["cuda"]["sm_evidence"][0]["sm"] = "75" + self.assert_invalid(mutant, "unsupported CUDA SM") + + def test_cuda_fallback_and_aot_state_rules_are_independently_live(self) -> None: + manifest = self.generated(cuda_facts()) + unavailable = PRIMARY_SMS.index("87") + fallback = copy.deepcopy(manifest) + fallback["cuda"]["sm_evidence"][unavailable]["portable_fallback"] = False + self.assertTrue( + any("portable fallback" in error for error in self.tool._cuda_policy(fallback)), + self.tool._cuda_policy(fallback), + ) + aot_state = copy.deepcopy(manifest) + aot_state["cuda"]["sm_evidence"][unavailable]["aot_evidence"] = evidence( + "absent" + ) + self.assertTrue( + any("unavailable AOT" in error for error in self.tool._cuda_policy(aot_state)), + self.tool._cuda_policy(aot_state), + ) + available = PRIMARY_SMS.index("80") + available_aot = copy.deepcopy(manifest) + available_aot["cuda"]["sm_evidence"][available]["aot_evidence"] = evidence( + "not-applicable" + ) + self.assertTrue( + any( + "available AOT" in error + for error in self.tool._cuda_policy(available_aot) + ), + self.tool._cuda_policy(available_aot), + ) + + def test_backend_channel_static_and_dependency_policies_fail_closed(self) -> None: + cpu = self.generated(cpu_facts()) + cuda = self.generated(cuda_facts()) + stable = copy.deepcopy(cpu) + stable["artifact"]["channel"] = "stable" + self.assert_invalid(stable, "stable") + for key in ("build", "archive_smoke", "dependency_audit", "runtime", "correctness"): + stable["evidence"][key] = evidence("passed") + self.assert_invalid(stable, "supply-chain") + for key in ("archive_checksum", "sbom", "provenance", "licenses"): + stable["supply_chain"][key] = evidence("passed") + stable["build"]["source_clean"] = False + self.assert_invalid(stable, "clean source") + stable["build"]["source_clean"] = True + self.assertEqual(self.tool.validate_manifest(stable, self.schema, ROOT), []) + for key in ("archive_checksum", "sbom", "provenance", "licenses"): + with self.subTest(stable_supply_chain=key): + mutant = copy.deepcopy(stable) + mutant["supply_chain"][key] = evidence("absent") + self.assert_invalid(mutant, key) + wrong_preview = copy.deepcopy(cuda) + wrong_preview["artifact"]["channel"] = "experimental-preview" + self.assert_invalid(wrong_preview, "channel") + musl_gpu = copy.deepcopy(cuda) + musl_gpu["artifact"]["id"] = "linux-x86_64-musl-cpu-static" + musl_gpu["artifact"]["static_boundary"] = "literal-static" + musl_gpu["artifact"]["channel"] = "experimental-preview" + musl_gpu["host"]["abi"] = "musl" + self.assert_invalid(musl_gpu, "musl") + bundled_driver = copy.deepcopy(cuda) + bundled_driver["dependencies"][0]["bundled"] = True + bundled_driver["dependencies"][0]["linkage"] = "static" + self.assert_invalid(bundled_driver, "driver") + bad_boundary = copy.deepcopy(cuda) + bad_boundary["backend"]["gpu_driver_boundary"] = "bundled" + self.assert_invalid(bad_boundary, "driver boundary") + bad_flags = copy.deepcopy(cpu) + bad_flags["backend"]["flags"]["VLLM_CPP_VULKAN"] = True + self.assert_invalid(bad_flags, "backend flags") + + def test_stable_runtime_and_correctness_rules_are_independently_live(self) -> None: + stable = self.generated(cpu_facts()) + stable["artifact"]["channel"] = "stable" + for key in ("build", "archive_smoke", "dependency_audit", "correctness"): + stable["evidence"][key] = evidence("passed") + for key in ("archive_checksum", "sbom", "provenance", "licenses"): + stable["supply_chain"][key] = evidence("passed") + runtime_errors = self.tool._publication_policy(stable) + self.assertTrue(any("runtime" in error for error in runtime_errors), runtime_errors) + stable["evidence"]["runtime"] = evidence("passed") + stable["evidence"]["correctness"] = evidence("absent") + correctness_errors = self.tool._publication_policy(stable) + self.assertTrue( + any("correctness" in error for error in correctness_errors), + correctness_errors, + ) + + def test_preview_publication_requires_clean_staged_archive_evidence(self) -> None: + preview = self.generated(cpu_facts()) + self.assertEqual(self.tool.validate_manifest(preview, self.schema, ROOT), []) + for key in ("build", "archive_smoke", "dependency_audit"): + with self.subTest(preview_evidence=key): + mutant = copy.deepcopy(preview) + mutant["evidence"][key] = evidence("absent") + self.assert_invalid(mutant, key) + dirty = copy.deepcopy(preview) + dirty["build"]["source_clean"] = False + self.assert_invalid(dirty, "clean source") + for key in ("runtime", "correctness", "performance"): + expected = "not-applicable" if key == "runtime" else "absent" + self.assertEqual(preview["evidence"][key]["state"], expected) + for key in ("archive_checksum", "sbom", "provenance", "licenses"): + self.assertEqual(preview["supply_chain"][key]["state"], "absent") + + def test_cuda_manifest_requires_the_nvidia_driver_dependency(self) -> None: + manifest = self.generated(cuda_facts()) + manifest["dependencies"] = [ + dependency + for dependency in manifest["dependencies"] + if dependency["name"] != "nvidia-driver" + ] + errors = self.tool.validate_manifest(manifest, self.schema, ROOT) + self.assertIn( + "$.dependencies: cuda requires external nvidia-driver declaration", + errors, + ) + + def test_backend_dependency_and_static_boundaries_are_fail_closed(self) -> None: + cpu = self.generated(cpu_facts()) + musl = copy.deepcopy(cpu) + musl["artifact"].update({ + "id": "linux-x86_64-musl-cpu-static", + "channel": "experimental-preview", + "static_boundary": "literal-static", + }) + musl["host"].update({"abi": "musl", "abi_version": "1.2.5"}) + musl["dependencies"] = [{ + "name": "musl", "version": "1.2.5", "kind": "library", + "linkage": "static", "bundled": True, "role": "runtime", + }] + self.assertEqual(self.tool.validate_manifest(musl, self.schema, ROOT), []) + dynamic_musl = copy.deepcopy(musl) + dynamic_musl["dependencies"][0].update({ + "linkage": "dynamic", "bundled": False, + }) + self.assert_invalid(dynamic_musl, "literal-static") + + cpu_driver = copy.deepcopy(cpu) + cpu_driver["dependencies"].append({ + "name": "synthetic-gpu-driver", "version": "1", "kind": "driver", + "linkage": "external", "bundled": False, "role": "external-runtime", + }) + self.assert_invalid(cpu_driver, "CPU") + + cuda = self.generated(cuda_facts()) + wrong_cuda_driver = copy.deepcopy(cuda) + wrong_cuda_driver["dependencies"][0]["name"] = "arbitrary-driver" + self.assert_invalid(wrong_cuda_driver, "NVIDIA") + + def accelerator( + artifact_id: str, + backend_name: str, + host: dict[str, str], + dependencies: list[dict[str, object]], + ) -> dict[str, object]: + manifest = copy.deepcopy(cpu) + manifest["artifact"]["id"] = artifact_id + manifest["host"].update(host) + manifest["backend"].update({ + "name": backend_name, + "flags": flags(backend_name), + "gpu_driver_boundary": "external-host-never-bundled", + }) + if backend_name == "mlx": + manifest["backend"]["flags"]["MLX_ROOT"] = "/opt/mlx" + manifest["build"]["resolved_cmake_options"] = copy.deepcopy( + manifest["backend"]["flags"] + ) + manifest["dependencies"] = dependencies + del manifest["cpu"] + return manifest + + vulkan = accelerator( + "linux-x86_64-glibc-vulkan", "vulkan", {}, [ + {"name": "vulkan-loader", "version": "1.3", "kind": "library", "linkage": "external", "bundled": False, "role": "external-runtime"}, + {"name": "vulkan-icd", "version": "1.3", "kind": "library", "linkage": "external", "bundled": False, "role": "external-runtime"}, + {"name": "vulkan-driver", "version": "tested-host", "kind": "driver", "linkage": "external", "bundled": False, "role": "external-runtime"}, + ], + ) + self.assertEqual(self.tool.validate_manifest(vulkan, self.schema, ROOT), []) + for name in ("vulkan-loader", "vulkan-icd", "vulkan-driver"): + mutant = copy.deepcopy(vulkan) + mutant["dependencies"] = [ + dep for dep in mutant["dependencies"] if dep["name"] != name + ] + self.assert_invalid(mutant, name) + mutant = copy.deepcopy(vulkan) + dependency = next( + dep for dep in mutant["dependencies"] if dep["name"] == name + ) + dependency.update({ + "linkage": "dynamic", "bundled": True, "role": "runtime", + }) + self.assert_invalid(mutant, name) + + frameworks = [ + {"name": name, "version": "macOS-system", "kind": "framework", "linkage": "external", "bundled": False, "role": "external-runtime"} + for name in ("Metal.framework", "Foundation.framework") + ] + metal = accelerator( + "macos-arm64-metal", "metal", + {"os": "macos", "arch": "aarch64", "abi": "macos", "abi_version": "14"}, + frameworks, + ) + self.assertEqual(self.tool.validate_manifest(metal, self.schema, ROOT), []) + for name in ("Metal.framework", "Foundation.framework"): + mutant = copy.deepcopy(metal) + mutant["dependencies"] = [ + dep for dep in mutant["dependencies"] if dep["name"] != name + ] + self.assert_invalid(mutant, name) + mutant = copy.deepcopy(metal) + dependency = next( + dep for dep in mutant["dependencies"] if dep["name"] == name + ) + dependency.update({ + "linkage": "dynamic", "bundled": True, "role": "runtime", + }) + self.assert_invalid(mutant, name) + + mlx_dependencies = frameworks + [ + {"name": "libmlx.dylib", "version": "0.29.1", "kind": "library", "linkage": "dynamic", "bundled": True, "role": "runtime"}, + {"name": "mlx.metallib", "version": "0.29.1", "kind": "library", "linkage": "dynamic", "bundled": True, "role": "runtime"}, + ] + mlx = accelerator( + "macos-arm64-metal-mlx", "mlx", + {"os": "macos", "arch": "aarch64", "abi": "macos", "abi_version": "14"}, + mlx_dependencies, + ) + self.assert_invalid(mlx, "license") + mlx["supply_chain"]["licenses"] = evidence("passed") + self.assertEqual(self.tool.validate_manifest(mlx, self.schema, ROOT), []) + unversioned_mlx = copy.deepcopy(mlx) + unversioned_mlx["dependencies"][2]["version"] = "unknown" + self.assert_invalid(unversioned_mlx, "versioned") + arbitrary_bundle = copy.deepcopy(mlx) + arbitrary_bundle["dependencies"].append({ + "name": "arbitrary.dylib", "version": "1", "kind": "library", + "linkage": "dynamic", "bundled": True, "role": "runtime", + }) + self.assert_invalid(arbitrary_bundle, "bundled") + + def test_cuda_feature_resolution_is_derived_from_current_cmake_table(self) -> None: + manifest = self.generated(cuda_facts()) + names = [feature["name"] for feature in manifest["backend"]["resolved_features"]] + self.assertEqual(names, [ + "fp4-mma", "cutlass-nvfp4", "cutlass-nvfp4-sm100", "cutlass-fp8", + "scaledmm-c3x-sm90", "scaledmm-c3x-sm100", "marlin-nvfp4", "fa2", + ]) + mutant = copy.deepcopy(manifest) + mutant["backend"]["resolved_features"][0]["compiled_sms"] = ["121a"] + self.assert_invalid(mutant, "resolved CUDA feature table") + + +if __name__ == "__main__": + unittest.main()