From 0adaff29f0d07455a2fe54ec439db6054ae8422c Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 15:37:43 +0100 Subject: [PATCH 1/7] chore(cpu): claim BACKEND-CPU Raspberry Pi 5 lane FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] From 7ff89feee5213aefcc4f93ae7d2e858861fde2a1 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 16:15:08 +0100 Subject: [PATCH 2/7] docs(cpu): spike Raspberry Pi 5 Cortex-A76 campaign Pin the Q8_K_XL vehicle, observed Pi hardware and PMU surface, recursive optimization method, correctness gates, and same-file llama.cpp floor before implementation. Record Pi evidence as pending without changing the binding 20-core Arm result. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 3 +- .agents/backend-matrix.md | 2 +- .agents/feature-matrix.md | 2 +- .../specs/rpi5-cortex-a76-cpu-optimization.md | 224 ++++++++++++++++++ .agents/state.md | 30 +++ docs/BENCHMARKS.md | 9 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- 8 files changed, 268 insertions(+), 6 deletions(-) create mode 100644 .agents/specs/rpi5-cortex-a76-cpu-optimization.md diff --git a/.agents/NOW.md b/.agents/NOW.md index fbee0e881..b8918da1a 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,8 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity | **TERMINAL (`QUANT-CT-MXFP4-FINAL-STACK`)**: c1 1.020x PASS + mem 2.63x; c2-c8 0.962-0.969 GPU-intrinsic. Both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, green, 32B strict char-identical); glue folds via `vt::FusedChain`, residual out-of-catalog Inductor GEMM-epilogue fusion (#46). `VT_MARLIN_DENSE` on | Record; branch not merged | +| RPi5 A76 CPU | **SPIKED PR #65**, no i8mm; all Pi evidence pending | PMU harness, then Qwen3.5-2B Q8 bring-up | +| MXFP4 parity | **TERMINAL:** c1 1.020x pass; c2-c8 0.962-0.969 GPU-intrinsic; final cap/glue levers exhausted | Record; branch unmerged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4), diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index e8da7ae62..149868082 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -223,7 +223,7 @@ on every listed target. | `BACKEND-ABI-VT` | Backend registry, device/queue, capture, drop-in adapter ABI | platform contract `vllm/platforms/interface.py:67-229`; stable stream `csrc/libtorch_stable/torch_utils.h:76-82`; raw Marlin boundary `csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu:344-356` | queue ID [device.h:9](../include/vt/device.h#L9), explicit resources [backend.h:53](../include/vt/backend.h#L53), scalar/layout ABI [ops.h:13](../include/vt/ops.h#L13), CUDA workspace/raw probe [cuda_dropin.cu:104](../src/vt/cuda/cuda_dropin.cu#L104) | [ported ABI tests](../tests/vt/test_dropin_abi.cpp#L86); CPU 94/94 + repair-focused 1/1; GCC13 repair at [test_dropin_abi.cpp:245](../tests/vt/test_dropin_abi.cpp#L245); exact sm_121a all-target build 100%, focused CUDA/ABI CTest 2/2, sanitizer 9/9 + 196/196 with 0 errors/leaks, 35B/27B gates 2/2 (`1141b79`, evidence manifest `4adbe952…601`) | [drop-in ABI](specs/dropin-kernel-abi.md) | `ANCHOR-BACKFILL` - sm_80/sm_90a cross-build and unchanged-trace/model A/B-memory proof remain; common scalar forwarder and legacy backend-shim migration remain; no production family migrated | CLAIM-BACKEND-ABI-W0-GPU-1 | | `BACKEND-ATTN-REGISTRY` | Attention-backend registry + platform-driven priority selection (extensibility **item 4** — the second portability seam, now REALIZED). Backends SELF-REGISTER per `(DeviceType, name)`; `Platform::get_attn_backend_priority()` advertises a capability-ordered name list; the selector returns the first REGISTERED name. Selection is DATA (register + priority slot), not an inline code edit. The concrete attention KERNEL stays at the vt:: op-table (`vt::PagedAttention`→`GetOp`), already device-additive | `vllm/v1/attention/backends/registry.py` (self-registration), `vllm/platforms/cuda.py:361-470` (`get_valid_backends`/`get_attn_backend_cls`) + `:84-166` (`_get_backend_priorities`, non-MLA major-10 vs else), `cpu.py:75-87` (CPU_ATTN) @ pin `e24d1b24` | registry+selector [registry.h:44](../include/vllm/v1/attention/registry.h#L44), [registry.cpp:60](../src/vllm/v1/attention/registry.cpp#L60); priority [interface.h:92](../include/vllm/platforms/interface.h#L92), CUDA major-10/else [cuda.cpp:44](../src/vllm/platforms/cuda.cpp#L44), CPU [cpu.cpp:29](../src/vllm/platforms/cpu.cpp#L29); FLASH_ATTN self-reg [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), GDN [gdn_attn.cpp:153](../src/vllm/v1/attention/backends/gdn_attn.cpp#L153) | [registry tests](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L61) (self-register + Make/throw + CUDA/CPU priority order + first-registered walk + override; 8 cases/25 assertions) + [platform priority](../tests/vllm/platforms/test_platform.cpp#L53); clean CPU `-Werror` + full CPU CTest green; behavior-preserving ⇒ same FA2 attention selected (**DGX-CONFIRMED @ `2c732e7`: 27B 235/235 + 35B 315/315 token-exact, FA2 sm_121a, memcheck 0/315**) | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) (item 4) | `ANCHOR-BACKFILL` — item-4 seam LANDED CPU + DGX-CONFIRMED; FLASHINFER/TRITON_ATTN/FLEX/TURBOQUANT are named in the priority (fidelity) but unimplemented ⇒ walk falls through to FLASH_ATTN (the behavior-preserving outcome); MLA-branch priorities deferred until an MLA model ports | CLAIM-ATTN-REGISTRY-1 | | `BACKEND-PLATFORM` | Platform capability/memory-model seam (`is_cuda`/`is_cpu`, `is_unified_memory`, `has_device_capability`, `supported_dtypes`, `residency_policy`, `supports_graph_capture`; `get_attn_backend_priority` REALIZED by `BACKEND-ATTN-REGISTRY` item 4; **+ the S3 capability predicates `supports_fp8`/`cutlass_fp4_supported`/`opaque_attention_op`/`is_integrated_gpu`/`support_static_graph_mode`/`is_device_capability_family`, mirroring `interface.py:914,933,977,1058,441-476` + `nvfp4_utils.py:56`, added by `CLAIM-BACKEND-SEAM-S3-1` and consumed by the fp4/fp8/graph gates in `qwen3_5.cpp`**); self-registered per `DeviceType` via `CurrentPlatform()` | `vllm/platforms/interface.py:134-229` (`class Platform`), `:409-439` (get/has_device_capability), `:181-187` (supported_dtypes) @ pin `e24d1b24` | interface [interface.h:56](../include/vllm/platforms/interface.h#L56), registry/CurrentPlatform [platform.cpp:57](../src/vllm/platforms/platform.cpp#L57), CPU [cpu.cpp:11](../src/vllm/platforms/cpu.cpp#L11), CUDA [cuda.cpp:14](../src/vllm/platforms/cuda.cpp#L14); composes `vt::Backend` [backend.h:22](../include/vt/backend.h#L22) | [platform tests](../tests/vllm/platforms/test_platform.cpp#L28) (registration + CPU capabilities + has_device_capability), clean CPU `-Werror` + full CPU CTest; 7 memory-model/residency sites migrated (`runner.cpp`, `model_registry.cpp`, `qwen3_5.cpp`) — **per-tensor sites key on `GetPlatform(.device.type).is_cuda()` (the OBJECT's device), NOT the process-global `CurrentPlatform()`, which is accelerator-first and would mis-route a CPU queue/tensor on a GPU box (regression fixed 2026-07-18; red DGX CPU tests `test_platform`/`test_qwen27_dense_forward`); `CurrentPlatform()` is reserved for process-level "which accelerator" questions only**. DGX behavior-preserving model gates 27B 235/235 + 35B 315/315 CONFIRMED @ (CUDA -Werror-clean after a GCC13 dangling-pointer FP fix). **Item-2 residency_policy() CONSUMPTION LANDED (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`):** the host-free / load-stream / DevicePool-cap decisions in `qwen3_5.cpp` now READ `GetPlatform(.device.type).residency_policy()` (per-device) instead of an inline `device.type`/env gate — host-free `qwen3_5.cpp:BuildMoeMarlinResident` via `ShouldReleaseHostWeights`, load-stream `Qwen3_5Model::PrepareMarlinResident` via `ShouldInterleaveLoadStream` (both `interface.h`), DevicePool soft cap via `residency_policy().device_pool_cap_bytes` (`DBuf`/`DevicePool`). `CudaPlatform::residency_policy().release_host_weights_after_upload` flipped **false→true** (now CONSUMED ⇒ reproduces today's GB10 host-free-after-Marlin-build EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH gate; `VT_MOE_HOST_FREE`/`VT_MOE_LOADSTREAM` env stay as overrides. Consumption unit-tested `test_platform.cpp` (7 cases/43 assertions). A new (discrete) GPU sets `residency_policy()` values ⇒ ZERO model edit. **DGX gate PASSED @ `62fc0e0` (production flags CUTLASS sm120a + FA2 sm_121a + Triton AOT, one flock): clean CUDA `-Werror` 0 warn, 27B 235/235 + 35B 315/315 token-exact, 35B VmHWM ≈ 4.0 GiB (load-stream win preserved), memcheck 0 errors.** | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) | `ANCHOR-BACKFILL` — item-1 extraction DONE + DGX-CONFIRMED; item-2 residency consumption LANDED + DGX-PROVEN (`CLAIM-BACKEND-PLATFORM-2`); kernel-shape dispatch branches deliberately left for items 4/5 | CLAIM-BACKEND-PLATFORM-1 / CLAIM-BACKEND-PLATFORM-2 | -| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass, but B4 speed/RSS and full model-context thread-safety gates remain open. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU backend was SILENTLY BROKEN on macOS — `CMakeLists.txt` gated the registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped every static registrar and `test_backend` failed 5/7 with `vt: no backend registered for device type 0`. Fixed to `if(APPLE) -force_load elseif(UNIX) --whole-archive`; `test_backend` is now **7/7 (18/18)** on the M4 and the WHOLE tree builds `-Werror`-clean there. This was a Metal prerequisite but the CPU backend is the bigger beneficiary: macOS is now a usable CPU-tier target (108,952 portable-tier assertions pass on Apple arm64). Two macOS-only test gaps remain, both PRE-EXISTING and unrelated to any backend: `test_serve_low_tools` (Linux-only `os.sched_getaffinity` / `POSIX_FADV_DONTNEED` in the Python bench tooling) and `test_safetensors` (`MappingRssKb` reads `/proc/self/smaps`) — recorded as debt in [environment.md](environment.md), not fixed | [threadpool leaf](specs/gguf-cpu-threadpool.md) | `PARTIAL` | - | +| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane SPIKED 2026-08-06 (PR #65):** four-core A76 with DotProd and no i8mm; exact Q8_K_XL model/hash, PMU surface, C++-before-assembly rule, recursive kernel-to-model gates and llama.cpp floor are fixed in the leaf spec. Pi correctness, baseline and every speed number remain `PENDING`; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | | `BACKEND-CPU-ZEN` | AMD Zen CPU with ZenDNN/zentorch dispatch and weight prepack | `platforms/zen_cpu.py:12-32`; detection `platforms/__init__.py:153-192`; `tests/test_zen_cpu_platform_detection.py:8-37` | generic CPU backend only; no Zen-specific dispatch | - | [CUDA inventory](specs/cuda-architecture-inventory.md); leaf spike required | `INVENTORIED` | - | | `BACKEND-ROCM` | AMD ROCm/HIP | `platforms/__init__.py:110-128`, `platforms/rocm.py:43-125`, ROCm ops rooted at `csrc/rocm/moe_q_gemm_rdna3.cu:1` | Lands through the realized seams: a `platforms/rocm.cpp` Platform + a `kROCM` attention TU self-registering via the [attn-registry seam](../include/vllm/v1/attention/registry.h#L44) + a `rocm.cpp` `get_attn_backend_priority()` slot (mirror `rocm.py`'s ROCM_ATTN/AITER ordering); ZERO selector/model/runner edit | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-XPU` | Intel XPU loyal port; gating exploration E4 ([backends.md](backends.md) kernel sourcing: SYCL-vs-Level-Zero call + whether upstream XPU attention contracts translate 1:1; explore when T2 scheduling begins) | `platforms/__init__.py:131-150`, `platforms/xpu.py:103-125` | enum slot only [device.h:11](../include/vt/device.h#L11) | [unavailable-backend test](../tests/vt/test_backend.cpp#L37); **SPIKE-ONLY verdict, MEASURED 2026-07-22: HW-BLOCKED — no Intel GPU on any box we own, and no acquisition path recorded.** Compounding finding: the loyalty target is INCOMPLETE upstream — `vllm/platforms/xpu.py:11-13` imports the EXTERNAL `vllm_xpu_kernels._C`/`._moe_C`/`._xpu_C`, and a search of the pinned tree for `*.sycl`/`*.dp.cpp` returns NOTHING, so there is no in-tree SYCL kernel source to mirror 1:1. What stays doable with zero Intel hardware: the attention-selector/priority POLICY port (pure data, unit-testable), oneAPI DPC++ compile coverage, and SYCL kernels executed on the oneAPI OpenCL CPU device for unit-level numerics. **No e2e or performance gate is proposed, because none can be run.** No XPU backend code exists | [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `SPIKE` | `CLAIM-BACKEND-FANOUT-1` | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index b5fea204a..f23ed19ba 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -274,7 +274,7 @@ evidence. |---|---|---|---|---| | `BACKEND-CUDA-SM121` | GB10/sm121a | `PARTIAL` | gate workload built, traced, token/perf gated; full component-family coverage is open | [backend row](backend-matrix.md#cuda-target-rows) | | `BACKEND-CUDA-OTHER` | vLLM sm70/75/80/86/87/89/90/100/101/103/110/120 targets | `ACTIVE` | 9 CUDA arches build-supported (sm80/86/87/89/90a/100a/103a/110/120a, single-arch portable-kernels-only, `-Werror` clean, SASS emitted); sm70/75/101 not build-supported; no non-121a target is runtime-validated | [backend matrix](backend-matrix.md), [CUDA inventory](specs/cuda-architecture-inventory.md) | -| `BACKEND-CPU` | production CPU | `PARTIAL` | persistent threadpool + chunked GEMM/row dispatch is 1/3/20-thread bit-identical and TSAN-clean; idle-host performance/RSS gate and compute-in-quant remain open | [backend matrix](backend-matrix.md) | +| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 (DotProd, no i8mm) is spiked under PR #65, with bring-up and all performance evidence pending | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md) | | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) | | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) | | `BACKEND-VULKAN` | Vulkan | `INVENTORIED` | runtime absent | [backend matrix](backend-matrix.md) | diff --git a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md new file mode 100644 index 000000000..e5ff72ad7 --- /dev/null +++ b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md @@ -0,0 +1,224 @@ +# BACKEND-CPU leaf spike: Raspberry Pi 5 Cortex-A76 optimization campaign + +Date: 2026-08-06 + +Row: `BACKEND-CPU` + +Claim: draft PR #65, branch `row/BACKEND-CPU` + +State at spike: `PARTIAL`, Raspberry Pi 5 correctness and speed `GATING` + +## Goal and scope + +Run `Qwen3.5-2B-UD-Q8_K_XL.gguf` correctly on a Raspberry Pi 5, then +systematically optimize every CPU loop reached by that fixed workload. Begin +with portable C++, proceed to Cortex-A76-tuned C++/NEON/SDOT, and add AArch64 +assembly only where compiler output and hardware counters prove a remaining +instruction-selection, scheduling, register-allocation, spill, addressing or +branching defect. + +The campaign covers model load, prefill, decode, sampling, the CPU threadpool +and serving overhead. It does not optimize kernels that the fixed Qwen trace +never reaches, claim other Arm cores, or run vLLM on the Pi. The x86-64 pinned +vLLM/current-engine run supplies correctness goldens; same-file llama.cpp on +the Pi is the performance and memory floor. + +## Fixed vehicle and hardware + +- Model: `unsloth/Qwen3.5-2B-GGUF/Qwen3.5-2B-UD-Q8_K_XL.gguf`, 2.83 GB, + SHA-256 `a53988df91157d78acaf3c95e22db179d13f6236061bdb86576494dc99b1bc3b`. +- Target: Raspberry Pi 5 Model B Rev 1.1, four Cortex-A76 r4p1 cores, + 1.5-2.4 GHz, 64 KiB L1D + 64 KiB L1I and 512 KiB L2 per core, shared + 2 MiB L3, 64-byte cache line, 8 GiB RAM. +- ISA observed from Linux: AArch64, ASIMD/NEON, FP16, RDM and DotProd/SDOT. + FEAT_I8MM is absent, so the existing MMLA tier must remain unavailable. +- Kernel: Debian Raspberry Pi `6.18.34+rpt-rpi-2712`. The + `armv8_cortex_a76` PMU is exposed as perf event source type 9 for CPUs 0-3, + with cycles, retired/speculated instructions, frontend/backend stalls, + branch, TLB and L1/L2/L3/last-level cache events. +- Native toolchain observed: GCC 14.2, CMake 3.31.6, Ninja and binutils 2.44. + `perf` and LLVM are not initially installed; direct `perf_event_open` is the + binding harness path, with `perf`/`llvm-mca` optional cross-checks. + +## Upstream and dependency chain + +vLLM is the semantic oracle, not the Pi implementation source: + +- `${VLLM_SOURCE}` at `555967922`: `vllm/platforms/cpu.py` owns CPU platform + selection; `csrc/cpu/torch_bindings.cpp` is its compiled-op entry layer. +- Qwen3.5 model semantics and tests remain those already ported by the + `Qwen3_5ForConditionalGeneration`/dense GGUF route. No model-specific decode + driver or kernel is introduced here. + +The performance implementation reference is llama.cpp at the project pin +`237ad9b96`: + +- `ggml/src/ggml-cpu/ggml-cpu.c`: graph execution, worker pool, task + partitioning and tensor-trait dispatch. +- `ggml/src/ggml-cpu/quants.c` and `ggml/src/ggml-cpu/arch/arm/quants.c`: + portable and Arm quantized dot products. +- `ggml/src/ggml-cpu/repack.cpp`: CPU feature-driven Q8 repacking and + microkernel selection. +- `src/models/qwen35.cpp`: the same architecture/model path used for the + on-device competitor. + +Additional microkernel references are Arm Compute Library, KleidiAI and +XNNPACK. They are design evidence only: code is ported only when licensing, +semantics, layout and tests are reconciled explicitly. + +## Current implementation inventory + +The current production path is already shared and model-independent: + +| Surface | Current anchor | Pi disposition | +|---|---|---| +| Backend and memory | `src/vt/cpu/cpu_backend.cpp` | portable, bring-up first | +| Persistent workers | `src/vt/cpu/cpu_threadpool.cpp` | 1/2/4-core scaling and affinity profiling owed | +| Generic CPU ops | `src/vt/cpu/cpu_ops.cpp` | trace all reached loops, preserve shared op routing | +| F16/BF16/F32 GEMM | `src/vt/cpu/cpu_matmul_elem.cpp` | NEON exists; A76 schedule/roof analysis owed | +| Quant GEMM dispatcher | `src/vt/cpu/cpu_quant_gemm.cpp` | portable path works; Pi-specific dispatch absent | +| Portable quant dots | `src/vt/cpu/cpu_quant_dot.cpp` | correctness fallback | +| Arm quant dots | `src/vt/cpu/cpu_quant_dot_arm.cpp` | i8mm-only fast tier; inert on Pi | +| Q8 repack | `src/vt/cpu/cpu_quant_repack{,_arm}.cpp` | i8mm path inert; DotProd/SDOT alternative owed | +| Paged attention | `src/vt/cpu/cpu_paged_attn.cpp` | correctness and 1/2/4-core scaling owed | +| Qwen3.5 forward | `src/vllm/model_executor/models/qwen3_5*.cpp` | reuse unchanged; no Pi-private forward | +| Provider selection | `include/vt/op_provider.h`, `src/vt/op_provider.cpp` | register A76 variants here with same-binary fallback | +| Existing microbench | `examples/quant_gemm_bench/main.cpp` | generalize, retain legacy cases | + +Every operation observed in the Qwen trace is entered into the experiment +manifest before tuning. Expected families are quant and elementwise GEMM/GEMV, +activation quantization, GDN prefill/decode, paged attention, normalization, +RoPE, elementwise fusion, embedding/logits, sampling, threadpool barriers and +loader page/repack work. The trace, not this expectation, is binding. + +## Benchmark harness contract + +Generalize the quant-only evidence tool into `vllm-cpu-kernel-bench` with: + +- `--op`, `--dtype`, `--m`, `--n`, `--k`, `--threads`, `--variant`, + `--warmup`, `--iterations`, `--cache={hot,l2,l3,stream}` and + `--format={text,json}` controls; +- deterministic synthetic inputs plus exact shape/layout fixtures captured + from the Qwen workload; +- batched timing that keeps timer/syscall cost below 0.1% of measured work; +- direct Linux `perf_event_open`, grouped so cycles/instructions accompany + each category pass and multiplexing is reported rather than hidden; +- time, cycles, instructions, IPC, frontend/backend stalls, branch misses, + cache/TLB events, bytes, effective bandwidth, checksum, compiler/flags, + CPU identity, affinity, frequency, temperature and throttling metadata; +- scopes for kernel, `vt::` operation, transformer block, full prefill, full + decode and serving request, so every accepted micro-change is checked in + its enclosing scopes. + +Runs pin one worker to one core for single-thread evidence and use CPUs 0-3 for +four-thread evidence. A run is void if the task migrates, the PMU group is +unaccounted-for, throttling is reported, frequency is unstable, another heavy +process overlaps, or output checksums differ. + +## Optimization procedure + +For each reached hot loop, build a measured speed-limit model before editing: + +1. frontend fetch/decode and branch prediction; +2. dispatch/uop and execution-pipe pressure; +3. loop-carried dependencies, accumulator count and scheduler/ROB occupancy; +4. register pressure, spills and address-generation work; +5. load/store issue width, cache-line crossings, cache/TLB misses and DRAM + bandwidth; +6. synchronization, false sharing, work partition and memory-bandwidth scaling. + +Test algorithm/layout/fusion/recomputation first, then blocking, alignment, +load width, unrolling, multiple accumulators, software pipelining, load/compute +interleaving and prefetch distance. GCC optimized/missed reports and disassembly +explain compiler behavior; `llvm-mca` is advisory; Pi PMU evidence is binding. + +An intrinsic variant is promoted to assembly only after it plateaus and the +compiler defect is demonstrated. Assembly lives in a separate `.S` file, +follows AAPCS64 and CFI rules, has no core-loop stack spill unless measured and +justified, handles complete aligned tiles, and declines to the tested C++ path +for unsupported features, shapes, alignment and tails. + +## Runtime dispatch and compatibility + +- Publish Linux CPU capability bits before first CPU op resolution. +- Detect DotProd through HWCAP. Detect the Cortex-A76 part before selecting a + schedule specialized for that microarchitecture. +- Register optimized implementations through `vt::OpProvider` at a higher + priority than `vt-native`; a call that cannot serve its dtype/shape declines + to the cached fallback. +- Keep a same-binary provider-disable control for every measured A/B. +- Never compile the whole library for Cortex-A76. Only provider translation + units receive `-mcpu=cortex-a76` or the minimum equivalent DotProd ISA flags, + preserving the generic AArch64 and x86 builds. + +## Tests to port and extend + +No new vLLM semantic test exists for CPU instruction scheduling. Existing +Qwen3.5 model/parity tests remain the semantic spec. Port and extend the +matching llama.cpp quant/repack cases for any borrowed layout or kernel: + +- exact portable-versus-provider results for integer dot/repack paths; +- existing tolerance contracts for floating reductions, without widening; +- M/N/K edge tiles, odd counts, short K, unaligned inputs and every tail; +- feature detection: A76+DotProd selects SDOT, A76 without DotProd and + non-A76 DotProd take their declared fallbacks, absent i8mm never executes + MMLA; +- deterministic results at 1/2/4 threads and concurrent submitter coverage; +- x86 and generic AArch64 builds, ASan/UBSan and thread sanitizer where + supported; +- disassembly audit proving the optimized object contains only its declared + ISA and assembly hot-loop invariants. + +## Correctness and performance gates + +1. x86-64 captures fixed prompt IDs, seeds, sampling parameters, 16 greedy + tokens, selected logits and operation fixtures from the pinned vLLM oracle + and the current CPU implementation. +2. Pi portable build loads the exact-hash GGUF and matches 16/16 tokens before + any optimized provider is enabled. +3. Every provider matches its portable operation contract and the Pi full + model repeats the cross-architecture golden. +4. Microbench A/B uses randomized interleaved trials. A retained metric-level + lever must improve its causal counter with no enclosing-scope regression; + it stays non-default until combined evidence improves an enclosing scope. +5. Full-model Pi gates use at least three interleaved repetitions for prefill, + decode, load time and peak memory at 1/2/4 threads. Same-file pinned + llama.cpp is measured on the same idle Pi. Concurrent serving follows after + the single-request gate. +6. Close an optimization family only after every inventoried lever category has + a win/neutral/negative/blocked disposition and the reconciled model predicts + less than 1% remaining end-to-end gain. A flat wall clock does not erase a + causal metric win; combinations are re-tested after each bottleneck moves. + +## Work breakdown + +| W | Item | Entry gate | Exit gate | +|---|---|---|---| +| R0 | Spike, Pi inventory and fixed model recipe | observed hardware facts | this spec + current record surfaces | +| R1 | General CPU kernel/PMU harness | existing quant bench | units, JSON schema, timer/counter self-tests | +| R2 | Portable Pi bring-up and x86 goldens | R0 | exact hash, load, 16/16 tokens, operation fixtures | +| R3 | Qwen trace + recursive scope profiling | R1-R2 | complete reached-loop inventory and binding baseline | +| R4 | A76 C++/NEON/SDOT providers | R3 ranked evidence | op correctness + causal metric win + no enclosing regression | +| R5 | A76 assembly candidates | R4 plateau + proven compiler gap | ABI/disassembly/correctness + recursive A/B | +| R6 | Whole-system/thread/serving exhaustion | accepted R4/R5 stack | all lever dispositions, <1% residual model, llama.cpp floor | + +R1-R3 are the first implementation checkpoint. R4/R5 split into separate +kernel-row PRs if the changed code exceeds the one-row helper size cap; this PR +does not silently absorb unrelated kernel families. + +## Risks and decisions + +- `BACKEND-CPU` already passes a 20-core Arm i8mm Qwen3.5-2B single-stream + llama.cpp floor. That evidence does not transfer to a four-core A76 without + i8mm; Pi numbers begin `PENDING` and never replace the existing scoreboard. +- The Q8_K_XL file contains substantial f16 weights, so quant GEMM alone cannot + be assumed dominant. A fresh trace ranks work before each lever. +- Reordered floating reductions may move near ties. Exact integer paths and the + current floating tolerance/token rules remain binding; speed never widens a + correctness threshold. +- PMU counter availability is verified, but permissions and event scheduling + can still reject an event. The harness reports unsupported events and never + fabricates zero counts. +- Temporary governor/tool installation and model transfer were approved by the + user. Every system setting is recorded and restored; no public service is + started and vLLM is not installed on the Pi. diff --git a/.agents/state.md b/.agents/state.md index 7eb2ebf2b..05593de39 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39275,3 +39275,33 @@ DISK-BLOCKED: NVFP4 working set (DiT + Qwen3-VL-32B NVFP4 encoder + 2 VAEs) is render. Comparability: vLLM-Omni CANNOT serve a quantized H3 on one GPU (BF16-only in practice; source-audited `a4ea67a2`) -> HW/loader-forced-indirect, DeepSeek-GGUF precedent. Draft PR is the claim. + + +## 2026-08-06T15:05 - BACKEND-CPU Raspberry Pi 5 Cortex-A76 campaign spiked + + +Draft PR #65 (`row/BACKEND-CPU`) records the target before implementation: +Qwen3.5-2B UD-Q8_K_XL, SHA-256 +`a53988df91157d78acaf3c95e22db179d13f6236061bdb86576494dc99b1bc3b`, on +the four-core Raspberry Pi 5 Cortex-A76. Read-only hardware inspection proved +ASIMD/FP16/RDM/DotProd and no i8mm, 64 KiB L1D/L1I + 512 KiB L2 per core, +shared 2 MiB L3, and a real `armv8_cortex_a76` Linux PMU exposing cycles, +instructions, frontend/backend stalls, branch, TLB and L1/L2/L3/LL-cache +events. The existing Arm MMLA/repack fast path is therefore inert on this +target; an SDOT route is a candidate, not yet a result. + +The committed leaf spec `.agents/specs/rpi5-cortex-a76-cpu-optimization.md` +pins the sequence: generalized PMU-backed kernel harness; x86 vLLM/current-CPU +goldens; portable Pi 16-token bring-up; trace-derived reached-loop inventory; +C++/NEON/SDOT experiments; assembly only after a measured compiler gap; and +recursive kernel→op→block→model/serving A/B after every accepted lever. A +causal PMU improvement may remain in the experimental stack before wall clock +moves, but no optimized provider becomes default without enclosing-scope +evidence and no area closes until every lever family has a disposition with +less than 1% predicted end-to-end residual. Same-file llama.cpp at project pin +`237ad9b96` is the Pi speed/memory floor; vLLM will not run on the Pi. + +No Pi model run, correctness result, throughput number or assembly kernel is +claimed at this checkpoint. `docs/BENCHMARKS.md` records `PENDING`; the prior +20-core Arm/i8mm binding result remains intact and explicitly does not transfer. +Next: R1 harness, then exact-hash model transfer and portable bring-up. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 5a7e8121e..666b45d46 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -8,7 +8,7 @@ | **vLLM** | Qwen3.6-35B-A3B NVFP4, GB10 | 0.93x to 1.03x: ahead at c4, worst c16 0.93x | identical | | **vLLM** | DeepSeek-V2-Lite (MLA), GB10 | 0.86x to 0.95x throughput, TTFT wins at c4/c8 | identical | | **vLLM** | Laguna-XS-2.1 NVFP4, GB10 | **parity+, 1.03x** (44.46 vs 43.10 tok/s, byte-exact, default config; bf16 weights now device-resident) | near-tie | -| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | prefill **1.18x ahead**, decode tie, memory parity | byte-identical | +| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill **1.18x ahead**, decode tie, memory parity. RPi5/A76: **PENDING** | byte-identical on binding arm; Pi pending | | **MLX-LM** | Qwen3-0.6B, Apple M4 | 97.6% warm total, prefill ahead | near-tie | | **DwarfStar** | DeepSeek-V4-Flash GGUF, GB10 | **beats ds4, 1.144x** (18.69 vs 16.33 tok/s, byte-exact, default config) | n/a, GGUF peer | @@ -154,6 +154,13 @@ host mirror is freed once the device Marlin resident is built. ## llama.cpp, CPU +Raspberry Pi 5 Cortex-A76 is a separate `PENDING` arm. It has four cores, +DotProd and no i8mm, so the binding 20-core Arm result below does not transfer. +Reproduction starts with the exact Q8_K_XL SHA-256 recorded in the +[RPi5 spike](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md), portable +16-token correctness, then three interleaved vllm.cpp/llama.cpp repetitions at +1/2/4 threads. No Pi throughput or memory number is accepted yet. + Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index b1a30412d..5674ff33e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86 AVX-512, Arm i8mm) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; Cortex-A76/SDOT gating) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 1daac77a0..5a7798257 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi (DotProd, no i8mm): spike complete; bring-up/PMU/perf `PENDING`. Concurrency open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | From 8ace554fdc7e0e427f5302b4c2c2bae75ac13a08 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 16:41:38 +0100 Subject: [PATCH 3/7] feat(cpu): add PMU-backed kernel benchmark harness Add a review-capped vt-op harness with deterministic quant GEMM fixtures, calibrated timing, cache-pressure profiles, JSON output, affinity and system metadata, plus grouped generic and Cortex-A76 perf_event_open counters. Keep unsupported and multiplexed events explicit and preserve the legacy quant benchmark. CPU-gated with GCC warning-as-error, clang-format, CTest CLI/schema checks, deterministic checksums, and direct 1-thread and 4-thread PMU runs. Pi execution and every performance claim remain pending. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 2 +- .agents/backend-matrix.md | 2 +- .agents/feature-matrix.md | 2 +- .agents/parity-ledger.md | 1 + .../specs/rpi5-cortex-a76-cpu-optimization.md | 5 +- .agents/state.md | 28 + docs/BENCHMARKS.md | 5 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- examples/CMakeLists.txt | 14 + examples/cpu_kernel_bench/main.cpp | 873 ++++++++++++++++++ tests/scripts/test_cpu_kernel_bench.py | 66 ++ 12 files changed, 994 insertions(+), 8 deletions(-) create mode 100644 examples/cpu_kernel_bench/main.cpp create mode 100644 tests/scripts/test_cpu_kernel_bench.py diff --git a/.agents/NOW.md b/.agents/NOW.md index b8918da1a..1f1dd9c84 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| RPi5 A76 CPU | **SPIKED PR #65**, no i8mm; all Pi evidence pending | PMU harness, then Qwen3.5-2B Q8 bring-up | +| RPi5 A76 CPU | **R1 HARNESS CPU-GREEN PR #65**; Pi/model evidence pending | Run harness on Pi, then Qwen3.5-2B Q8 bring-up | | MXFP4 parity | **TERMINAL:** c1 1.020x pass; c2-c8 0.962-0.969 GPU-intrinsic; final cap/glue levers exhausted | Record; branch unmerged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 149868082..7926b7310 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -223,7 +223,7 @@ on every listed target. | `BACKEND-ABI-VT` | Backend registry, device/queue, capture, drop-in adapter ABI | platform contract `vllm/platforms/interface.py:67-229`; stable stream `csrc/libtorch_stable/torch_utils.h:76-82`; raw Marlin boundary `csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu:344-356` | queue ID [device.h:9](../include/vt/device.h#L9), explicit resources [backend.h:53](../include/vt/backend.h#L53), scalar/layout ABI [ops.h:13](../include/vt/ops.h#L13), CUDA workspace/raw probe [cuda_dropin.cu:104](../src/vt/cuda/cuda_dropin.cu#L104) | [ported ABI tests](../tests/vt/test_dropin_abi.cpp#L86); CPU 94/94 + repair-focused 1/1; GCC13 repair at [test_dropin_abi.cpp:245](../tests/vt/test_dropin_abi.cpp#L245); exact sm_121a all-target build 100%, focused CUDA/ABI CTest 2/2, sanitizer 9/9 + 196/196 with 0 errors/leaks, 35B/27B gates 2/2 (`1141b79`, evidence manifest `4adbe952…601`) | [drop-in ABI](specs/dropin-kernel-abi.md) | `ANCHOR-BACKFILL` - sm_80/sm_90a cross-build and unchanged-trace/model A/B-memory proof remain; common scalar forwarder and legacy backend-shim migration remain; no production family migrated | CLAIM-BACKEND-ABI-W0-GPU-1 | | `BACKEND-ATTN-REGISTRY` | Attention-backend registry + platform-driven priority selection (extensibility **item 4** — the second portability seam, now REALIZED). Backends SELF-REGISTER per `(DeviceType, name)`; `Platform::get_attn_backend_priority()` advertises a capability-ordered name list; the selector returns the first REGISTERED name. Selection is DATA (register + priority slot), not an inline code edit. The concrete attention KERNEL stays at the vt:: op-table (`vt::PagedAttention`→`GetOp`), already device-additive | `vllm/v1/attention/backends/registry.py` (self-registration), `vllm/platforms/cuda.py:361-470` (`get_valid_backends`/`get_attn_backend_cls`) + `:84-166` (`_get_backend_priorities`, non-MLA major-10 vs else), `cpu.py:75-87` (CPU_ATTN) @ pin `e24d1b24` | registry+selector [registry.h:44](../include/vllm/v1/attention/registry.h#L44), [registry.cpp:60](../src/vllm/v1/attention/registry.cpp#L60); priority [interface.h:92](../include/vllm/platforms/interface.h#L92), CUDA major-10/else [cuda.cpp:44](../src/vllm/platforms/cuda.cpp#L44), CPU [cpu.cpp:29](../src/vllm/platforms/cpu.cpp#L29); FLASH_ATTN self-reg [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), GDN [gdn_attn.cpp:153](../src/vllm/v1/attention/backends/gdn_attn.cpp#L153) | [registry tests](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L61) (self-register + Make/throw + CUDA/CPU priority order + first-registered walk + override; 8 cases/25 assertions) + [platform priority](../tests/vllm/platforms/test_platform.cpp#L53); clean CPU `-Werror` + full CPU CTest green; behavior-preserving ⇒ same FA2 attention selected (**DGX-CONFIRMED @ `2c732e7`: 27B 235/235 + 35B 315/315 token-exact, FA2 sm_121a, memcheck 0/315**) | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) (item 4) | `ANCHOR-BACKFILL` — item-4 seam LANDED CPU + DGX-CONFIRMED; FLASHINFER/TRITON_ATTN/FLEX/TURBOQUANT are named in the priority (fidelity) but unimplemented ⇒ walk falls through to FLASH_ATTN (the behavior-preserving outcome); MLA-branch priorities deferred until an MLA model ports | CLAIM-ATTN-REGISTRY-1 | | `BACKEND-PLATFORM` | Platform capability/memory-model seam (`is_cuda`/`is_cpu`, `is_unified_memory`, `has_device_capability`, `supported_dtypes`, `residency_policy`, `supports_graph_capture`; `get_attn_backend_priority` REALIZED by `BACKEND-ATTN-REGISTRY` item 4; **+ the S3 capability predicates `supports_fp8`/`cutlass_fp4_supported`/`opaque_attention_op`/`is_integrated_gpu`/`support_static_graph_mode`/`is_device_capability_family`, mirroring `interface.py:914,933,977,1058,441-476` + `nvfp4_utils.py:56`, added by `CLAIM-BACKEND-SEAM-S3-1` and consumed by the fp4/fp8/graph gates in `qwen3_5.cpp`**); self-registered per `DeviceType` via `CurrentPlatform()` | `vllm/platforms/interface.py:134-229` (`class Platform`), `:409-439` (get/has_device_capability), `:181-187` (supported_dtypes) @ pin `e24d1b24` | interface [interface.h:56](../include/vllm/platforms/interface.h#L56), registry/CurrentPlatform [platform.cpp:57](../src/vllm/platforms/platform.cpp#L57), CPU [cpu.cpp:11](../src/vllm/platforms/cpu.cpp#L11), CUDA [cuda.cpp:14](../src/vllm/platforms/cuda.cpp#L14); composes `vt::Backend` [backend.h:22](../include/vt/backend.h#L22) | [platform tests](../tests/vllm/platforms/test_platform.cpp#L28) (registration + CPU capabilities + has_device_capability), clean CPU `-Werror` + full CPU CTest; 7 memory-model/residency sites migrated (`runner.cpp`, `model_registry.cpp`, `qwen3_5.cpp`) — **per-tensor sites key on `GetPlatform(.device.type).is_cuda()` (the OBJECT's device), NOT the process-global `CurrentPlatform()`, which is accelerator-first and would mis-route a CPU queue/tensor on a GPU box (regression fixed 2026-07-18; red DGX CPU tests `test_platform`/`test_qwen27_dense_forward`); `CurrentPlatform()` is reserved for process-level "which accelerator" questions only**. DGX behavior-preserving model gates 27B 235/235 + 35B 315/315 CONFIRMED @ (CUDA -Werror-clean after a GCC13 dangling-pointer FP fix). **Item-2 residency_policy() CONSUMPTION LANDED (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`):** the host-free / load-stream / DevicePool-cap decisions in `qwen3_5.cpp` now READ `GetPlatform(.device.type).residency_policy()` (per-device) instead of an inline `device.type`/env gate — host-free `qwen3_5.cpp:BuildMoeMarlinResident` via `ShouldReleaseHostWeights`, load-stream `Qwen3_5Model::PrepareMarlinResident` via `ShouldInterleaveLoadStream` (both `interface.h`), DevicePool soft cap via `residency_policy().device_pool_cap_bytes` (`DBuf`/`DevicePool`). `CudaPlatform::residency_policy().release_host_weights_after_upload` flipped **false→true** (now CONSUMED ⇒ reproduces today's GB10 host-free-after-Marlin-build EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH gate; `VT_MOE_HOST_FREE`/`VT_MOE_LOADSTREAM` env stay as overrides. Consumption unit-tested `test_platform.cpp` (7 cases/43 assertions). A new (discrete) GPU sets `residency_policy()` values ⇒ ZERO model edit. **DGX gate PASSED @ `62fc0e0` (production flags CUTLASS sm120a + FA2 sm_121a + Triton AOT, one flock): clean CUDA `-Werror` 0 warn, 27B 235/235 + 35B 315/315 token-exact, 35B VmHWM ≈ 4.0 GiB (load-stream win preserved), memcheck 0 errors.** | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) | `ANCHOR-BACKFILL` — item-1 extraction DONE + DGX-CONFIRMED; item-2 residency consumption LANDED + DGX-PROVEN (`CLAIM-BACKEND-PLATFORM-2`); kernel-shape dispatch branches deliberately left for items 4/5 | CLAIM-BACKEND-PLATFORM-1 / CLAIM-BACKEND-PLATFORM-2 | -| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane SPIKED 2026-08-06 (PR #65):** four-core A76 with DotProd and no i8mm; exact Q8_K_XL model/hash, PMU surface, C++-before-assembly rule, recursive kernel-to-model gates and llama.cpp floor are fixed in the leaf spec. Pi correctness, baseline and every speed number remain `PENDING`; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | +| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06 (PR #65):** R0 pins the exact Q8_K_XL hash, four-core A76 DotProd/no-i8mm target, recursive gates and llama.cpp floor. R1 [PMU harness](../examples/cpu_kernel_bench/main.cpp) is x86 CPU-gated: parameterized quant GEMM, deterministic checksum, calibrated batched timing, 1/2/4-core affinity, cache-pressure profiles, JSON, generic + A76 raw grouped counters with IPC/multiplex/unsupported status, frequency/temperature/governor/throttle metadata; [black-box CLI test](../tests/scripts/test_cpu_kernel_bench.py). Pi execution, correctness, baseline and every Pi speed number remain `PENDING`; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | | `BACKEND-CPU-ZEN` | AMD Zen CPU with ZenDNN/zentorch dispatch and weight prepack | `platforms/zen_cpu.py:12-32`; detection `platforms/__init__.py:153-192`; `tests/test_zen_cpu_platform_detection.py:8-37` | generic CPU backend only; no Zen-specific dispatch | - | [CUDA inventory](specs/cuda-architecture-inventory.md); leaf spike required | `INVENTORIED` | - | | `BACKEND-ROCM` | AMD ROCm/HIP | `platforms/__init__.py:110-128`, `platforms/rocm.py:43-125`, ROCm ops rooted at `csrc/rocm/moe_q_gemm_rdna3.cu:1` | Lands through the realized seams: a `platforms/rocm.cpp` Platform + a `kROCM` attention TU self-registering via the [attn-registry seam](../include/vllm/v1/attention/registry.h#L44) + a `rocm.cpp` `get_attn_backend_priority()` slot (mirror `rocm.py`'s ROCM_ATTN/AITER ordering); ZERO selector/model/runner edit | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-XPU` | Intel XPU loyal port; gating exploration E4 ([backends.md](backends.md) kernel sourcing: SYCL-vs-Level-Zero call + whether upstream XPU attention contracts translate 1:1; explore when T2 scheduling begins) | `platforms/__init__.py:131-150`, `platforms/xpu.py:103-125` | enum slot only [device.h:11](../include/vt/device.h#L11) | [unavailable-backend test](../tests/vt/test_backend.cpp#L37); **SPIKE-ONLY verdict, MEASURED 2026-07-22: HW-BLOCKED — no Intel GPU on any box we own, and no acquisition path recorded.** Compounding finding: the loyalty target is INCOMPLETE upstream — `vllm/platforms/xpu.py:11-13` imports the EXTERNAL `vllm_xpu_kernels._C`/`._moe_C`/`._xpu_C`, and a search of the pinned tree for `*.sycl`/`*.dp.cpp` returns NOTHING, so there is no in-tree SYCL kernel source to mirror 1:1. What stays doable with zero Intel hardware: the attention-selector/priority POLICY port (pure data, unit-testable), oneAPI DPC++ compile coverage, and SYCL kernels executed on the oneAPI OpenCL CPU device for unit-level numerics. **No e2e or performance gate is proposed, because none can be run.** No XPU backend code exists | [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `SPIKE` | `CLAIM-BACKEND-FANOUT-1` | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index f23ed19ba..ad078e39e 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -274,7 +274,7 @@ evidence. |---|---|---|---|---| | `BACKEND-CUDA-SM121` | GB10/sm121a | `PARTIAL` | gate workload built, traced, token/perf gated; full component-family coverage is open | [backend row](backend-matrix.md#cuda-target-rows) | | `BACKEND-CUDA-OTHER` | vLLM sm70/75/80/86/87/89/90/100/101/103/110/120 targets | `ACTIVE` | 9 CUDA arches build-supported (sm80/86/87/89/90a/100a/103a/110/120a, single-arch portable-kernels-only, `-Werror` clean, SASS emitted); sm70/75/101 not build-supported; no non-121a target is runtime-validated | [backend matrix](backend-matrix.md), [CUDA inventory](specs/cuda-architecture-inventory.md) | -| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 (DotProd, no i8mm) is spiked under PR #65, with bring-up and all performance evidence pending | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md) | +| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 (DotProd, no i8mm) R1 PMU harness is x86 CPU-gated under PR #65; Pi execution, model bring-up and all Pi performance evidence remain pending | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md), [harness](../examples/cpu_kernel_bench/main.cpp) | | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) | | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) | | `BACKEND-VULKAN` | Vulkan | `INVENTORIED` | runtime absent | [backend matrix](backend-matrix.md) | diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 49de1eb98..44ab5cff4 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -914,3 +914,4 @@ Columns: | 2026-08-06 (`row/KERNEL-FA2-GQA-SWAP`; `CLAIM-KERNEL-FA2-GQA-SWAP`; kernel `KERNEL-ATTN-FA2`; gated default-OFF, lifecycle unchanged) | Ports vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization into the d128 varlen decode launcher (`LaunchDecodeVarlenFA2Bf16`, gate `VT_FA2_DECODE_GQA_SWAP`): the Qwen3-dense decode grid becomes `(batch, kv_heads)` not `(batch, hq)` — the ngroups query heads pack into seqlen_q, KV read once/group, presented WITHOUT a materialized transpose via kv-major-group-minor strides (a 1:1 mirror of the already-shipped d256 `LaunchDecodeFA2Bf16` swap). OFF path byte-identical to the prior plain-varlen reduction; ON is non-byte-exact only when num_splits>1 (split reduction order → near-tie, toward vLLM's own numerics). | Mirrors `flash-attention @ 2c839c33` `mha_fwd_kvcache` seqlenq_ngroups_swapped + `set_params_splitkv` and vLLM v0.25.0 `flash_attn.py flash_attn_varlen_func` decode (#47 measured vLLM's swapped grid `(1,6,16)` = batch×kv_heads vs ours `(1,3,64)` = batch×query_heads). The vendored `flash_fwd_kernel.h` `get_lse_tile`/combine already honor the flag in both the num_splits==1 direct-write and >1 combine paths (the d256 arm is the proof). | GB10 sm_121a CUDA 13.0: op RED-first test 280/280 (both GQA ratios × batch{1,2,4,8} × short+long ctx; `swap_launches==1` proves the grid engaged; swap-vs-plain near-tie; MHA-inert) — RED proven (wrong swapped stride → 26,528 violations); full binary 28/28·454,679 no regression; compute-sanitizer 0-err/0-leak; #44 MXFP4 e2e smoke swap-ON 3/3 deterministic TOKEN-EXACT + coherent, byte-identical to swap-OFF. `benchmark_binding=false` (c1-c8 x3 re-bench + default flip = recorded next step; #47 projects flash ~28%@c2 / ~55%@c8 of the gap). | | 2026-08-06 (`row/KERNEL-MARLIN-DENSE-PORT`; `CLAIM-KERNEL-MARLIN-DENSE-PORT`; kernel `KERNEL-GEMM-MARLIN-W4A16`; gated default-OFF, lifecycle unchanged) | Vendors vLLM's OWN dense marlin W4A16 GEMM as a new `vt::MarlinDenseGemm` op (`VT_MARLIN_DENSE`, default OFF) and routes the E=1 dense NVFP4/MXFP4 projections (`dense_nvfp4_gemm.h` `MatmulNvfp4MarlinD`/`MatmulMxfp4W4A16D`/`GateUpFusedMarlinD`) through it. The dense kernel is direct-A + tile-per-CTA with vLLM's OWN dense fp32-C_tmp reduce, so at M<=8 it runs the sms-wide (48-CTA) grid WITHOUT the one-bf16-ULP shift the `VT_MARLIN_E1_PAR1` MoE-route par-regroup costs (#54: that ULP flips a strict 32B-NVFP4A16 token). Reuses the EXISTING marlin resident + workspace (same `marlin_permute` repack for dense and MoE — confirmed, no shim); rank-2 operand views, no moe_align gather. | 1:1 lift of vLLM @ `555967922` `csrc/libtorch_stable/quantization/marlin/`: `marlin.cu:326-541` (`marlin::marlin_mm` + config helpers) → `marlin_mm_dense.cu`; the torch::stable `marlin_gemm` wrapper (`:545-894`) → torch-free `cuda_marlin_dense.cu` launcher (mirrors `cuda_moe_marlin.cu`, dense c_tmp sizing `:713-716`); `kernel.h`/`marlin_template.h:1-2081` verbatim (the DENSE kernel — DISTINCT from the moe one, but SAME 12-param `Marlin<>` template so the generated `kernel_selector.h`+`sm80_*.cu` instantiation set is shared, namespace `marlin` from the local kernel.h). Shared `marlin.cuh`/`marlin_dtypes.cuh`/`dequant.h`/`marlin_mma.h` diff-verified byte-identical. Forced-Marlin a16 selection `kernels/linear/__init__.py:879-881`. | CPU `-fsyntax-only` CLEAN (`ops.cpp` + the `VT_MARLIN_NVFP4` routing header). GPU compile: all 3 new dense `.cu` compile CLEAN on dgx GB10 sm_121a under exact production flags (`-Werror=all-warnings`, `-static-global-template-stub=false`, `--generate-code=…sm_121a`). RED-first unit battery WRITTEN (`test_ops_moe_grouped.cpp`: NVFP4+MXFP4, M=1..8 × 3 shapes, dense-vs-CPU-ref AND dense-vs-grouped-route, row-shifted stride RED-injection). `benchmark_binding=false`; GPU EXEC gates (unit run + strict token battery dense-ON vs oracle incl. 32B-NVFP4A16:344 + launch-counter + nsys 48-CTA + binding c1..c8 x3) are the scoped dgx follow-up; default stays OFF until the strict battery proves oracle byte-match and the binding beats the MoE route (state `KERNEL-MARLIN-DENSE-PORT`). | | 2026-08-06 (`row/H3-FP4-SPEED`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; lifecycle unchanged) | **MiniMax-H3 W-FP4a — fp4-RESIDENT NVFP4 routing for the device DiT forward (NO new quant code).** Until now both NVFP4 loaders dequantized the packed FP4 projections to bf16 and the device forward ran `vt::MatmulBT`, so the sm_121a FP4 tensor-core route never ran for H3. Adds `Nvfp4Weight` carriers to `MiniMaxH3DitBlockWeights`/`MiniMaxH3DitWeights`, a fp4-resident streamer `StreamMiniMaxH3Nvfp4ToDeviceFp4` (keeps the compressed-tensors triple host-resident; the shared dispatcher uploads+repacks lazily then frees the fp4 originals, so peak device memory is ~1/4 of the bf16 arm), and a `LinearDev` dispatch that routes a non-Empty fp4 projection through `dense_nvfp4::MatmulNvfp4W4A16D`. | The routing is vLLM's OWN forced-Marlin-for-a16 selection: the checkpoint is weight-only NVFP4 (no `input_activations`, `IsTrueW4A4()==false`), so `kernels/linear/__init__.py:879-881` forces the Marlin W4A16 kernel, mirrored by `include/vllm/model_executor/models/dense_nvfp4_gemm.h:12-22,505-549` (`MatmulNvfp4W4A16D` -> single-expert `vt::MoeGroupedGemmNvfp4Marlin`, the SAME kernel Laguna routed-experts + dense Qwen3-32B NVFP4 use). Not cutlass-fp4/W4A4 (needs fp4 activations, private to `qwen3_5.cpp`). fc1 is already merged `[gate;up]` -> one W4A16 GEMM + `vt::SiluAndMul`. | **CPU-GATED (wiring), verified**: `test_minimax_h3` 62/62 cases / 30039 assertions, 0 failed. The synthetic-NVFP4 case streams the fp4 twin, asserts the loader kept the projections PACKED (fp4 slot set / bf16 slot Empty, and the inverse for the bf16 loader), runs fp4 + bf16 device forwards on the SAME file, asserts the W4A16 dispatcher executed ALL 11 quantized GEMMs (the `Nvfp4W4A16Stats` this-path-ran counter), and bounds fp4-vs-bf16 <= 2e-3. On CPU the dispatcher has no Marlin op so it falls to the bf16 arm's own dequant+matmul (hence a WIRING gate here); the Marlin kernel numerics are CUDA-gated independently by `test_ops_nvfp4_matmul` / `test_linear_method` (2e-3 f32-out / 8e-3 bf16-out vs a bf16 reference). `benchmark_binding=false`. PENDING: GB10 CUDA build + the fp4-vs-bf16 numeric delta and steady per-step timing (disk/build window); real-checkpoint t2va e2e DISK-BLOCKED (~41 GB working set). Comparability: vLLM-Omni serves NO quantized H3 (BF16-only in practice; source-audited `a4ea67a2`, spec §8.3) -> HW/loader-forced-indirect. | +| 2026-08-06 (`row/BACKEND-CPU`; `BACKEND-CPU` R1; PR #65; lifecycle remains `PARTIAL`) | Adds `vllm-cpu-kernel-bench`, a developer-only vt-op benchmark substrate: deterministic quant-GEMM fixtures, calibrated batched timing, cache-pressure profiles, affinity, JSON, checksums, system metadata, and grouped generic/Cortex-A76 `perf_event_open` counters with explicit multiplex/unsupported status. Production dispatch and numerics are unchanged. | No vLLM behavior counterpart; vLLM remains the x86 semantic oracle and llama.cpp `237ad9b96` remains the Pi performance floor. The quant fixture invokes the existing `vt::MatmulBTQuant` contract unchanged. | **CPU-GATED, `benchmark_binding=false`.** GCC 15.2 `-Wall -Wextra -Werror` build; clang-format clean; `test_cpu_kernel_bench_cli` deterministic JSON schema/checksum + invalid-input + structured-counter cases; direct 1/4-thread x86 runs and real generic PMU counts. X86 timings are non-binding tool validation. Pi PMU execution, model correctness, throughput and memory all remain `PENDING`. | diff --git a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md index e5ff72ad7..8603f671f 100644 --- a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md +++ b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md @@ -83,7 +83,8 @@ The current production path is already shared and model-independent: | Paged attention | `src/vt/cpu/cpu_paged_attn.cpp` | correctness and 1/2/4-core scaling owed | | Qwen3.5 forward | `src/vllm/model_executor/models/qwen3_5*.cpp` | reuse unchanged; no Pi-private forward | | Provider selection | `include/vt/op_provider.h`, `src/vt/op_provider.cpp` | register A76 variants here with same-binary fallback | -| Existing microbench | `examples/quant_gemm_bench/main.cpp` | generalize, retain legacy cases | +| CPU microbench | `examples/cpu_kernel_bench/main.cpp` | R1 vt-op/PMU substrate CPU-gated; Pi execution pending | +| Legacy quant evidence | `examples/quant_gemm_bench/main.cpp` | retained unchanged for prior results | Every operation observed in the Qwen trace is entered into the experiment manifest before tuning. Expected families are quant and elementwise GEMM/GEMV, @@ -195,7 +196,7 @@ matching llama.cpp quant/repack cases for any borrowed layout or kernel: | W | Item | Entry gate | Exit gate | |---|---|---|---| | R0 | Spike, Pi inventory and fixed model recipe | observed hardware facts | this spec + current record surfaces | -| R1 | General CPU kernel/PMU harness | existing quant bench | units, JSON schema, timer/counter self-tests | +| R1 | **CPU-GATED** general CPU kernel/PMU harness | existing quant bench | warning-clean build; JSON/CLI/timer/counter contract; 1/4-thread runs | | R2 | Portable Pi bring-up and x86 goldens | R0 | exact hash, load, 16/16 tokens, operation fixtures | | R3 | Qwen trace + recursive scope profiling | R1-R2 | complete reached-loop inventory and binding baseline | | R4 | A76 C++/NEON/SDOT providers | R3 ranked evidence | op correctness + causal metric win + no enclosing regression | diff --git a/.agents/state.md b/.agents/state.md index 05593de39..204c31d4c 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39305,3 +39305,31 @@ No Pi model run, correctness result, throughput number or assembly kernel is claimed at this checkpoint. `docs/BENCHMARKS.md` records `PENDING`; the prior 20-core Arm/i8mm binding result remains intact and explicitly does not transfer. Next: R1 harness, then exact-hash model transfer and portable bring-up. + + +## 2026-08-06T16:39 - BACKEND-CPU R1 PMU harness CPU-gated + + +`examples/cpu_kernel_bench/main.cpp` adds the review-capped +`vllm-cpu-kernel-bench` vt-op substrate without changing a production kernel. +The initial fixture covers `MatmulBTQuant` Q8_0/Q4_K/Q6_K with M/N/K/thread, +provider, warmup/iteration, hot/L2/L3/stream pressure and text/JSON controls. +Inputs are deterministic and every timing/counter pass must preserve the same +FNV checksum. Hot cases calibrate calls per sample until clock-read overhead is +below 0.1%; cold profiles expose an invalid timing instead of hiding overhead. + +Linux counters use `perf_event_open` directly. Generic passes and four +Cortex-A76 raw groups repeat cycles/instructions alongside stalls, branch, +L1/L2/LL-cache and TLB events; counts are scaled by enabled/running time and +each pass reports `ok`, `partial`, `multiplexed`, `unscheduled` or +`unsupported`. The JSON also records IPC, logical bytes/bandwidth, compiler +features, CPU/affinity/migration, frequency, governor, temperature and +Raspberry Pi `get_throttled` output when `vcgencmd` exists. + +GCC 15.2 CPU build passed `-Wall -Wextra -Werror`; clang-format is clean; +CTest `test_cpu_kernel_bench_cli` passed its deterministic JSON-schema, +invalid-input and structured-counter cases. Direct x86 executions passed at +1 and 4 threads, including a model-shaped Q8_0 decode fixture, and generic PMU +groups returned real counts. Those x86 timings are tool validation only and +are not binding performance evidence. Pi PMU execution, Qwen correctness and +llama.cpp comparison remain `PENDING`; next is R1-on-Pi, then R2 bring-up. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 666b45d46..9ec022f92 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -159,7 +159,10 @@ DotProd and no i8mm, so the binding 20-core Arm result below does not transfer. Reproduction starts with the exact Q8_K_XL SHA-256 recorded in the [RPi5 spike](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md), portable 16-token correctness, then three interleaved vllm.cpp/llama.cpp repetitions at -1/2/4 threads. No Pi throughput or memory number is accepted yet. +1/2/4 threads. The R1 vt-op harness is CPU-gated and emits PMU-accounted JSON; +the first Pi probe is +`vllm-cpu-kernel-bench --dtype q8_0 --m 1 --n 3072 --k 2048 --threads 1 --cache stream --counters a76 --format json`. +No Pi throughput or memory number is accepted yet. Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5674ff33e..7a9a743dd 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; Cortex-A76/SDOT gating) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; A76 PMU harness, SDOT gating) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 5a7798257..5a8936732 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi (DotProd, no i8mm): spike complete; bring-up/PMU/perf `PENDING`. Concurrency open | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi (DotProd, no i8mm): R1 PMU/JSON vt-op harness CPU-gated; Pi execution, model bring-up and perf `PENDING`. Concurrency open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index c3a40f23e..797b542cc 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -51,6 +51,20 @@ target_link_libraries(quant-gemm-bench PRIVATE vllm::vllm) target_include_directories(quant-gemm-bench PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_set_warnings(quant-gemm-bench) +# General PMU-backed CPU operation harness (BACKEND-CPU R1). quant-gemm-bench +# stays available so the earlier binding evidence remains reproducible. +add_executable(vllm-cpu-kernel-bench cpu_kernel_bench/main.cpp) +target_link_libraries(vllm-cpu-kernel-bench PRIVATE vllm::vllm) +target_include_directories(vllm-cpu-kernel-bench PRIVATE ${CMAKE_SOURCE_DIR}/src) +vllm_cpp_set_warnings(vllm-cpu-kernel-bench) +if(VLLM_CPP_BUILD_TESTS) + find_package(Python3 COMPONENTS Interpreter REQUIRED) + add_test(NAME test_cpu_kernel_bench_cli + COMMAND ${Python3_EXECUTABLE} + ${CMAKE_SOURCE_DIR}/tests/scripts/test_cpu_kernel_bench.py + $) +endif() + # The OpenAI HTTP server example is gated on the vendored cpp-httplib transport. if(VLLM_CPP_SERVER) add_executable(server server/main.cpp) diff --git a/examples/cpu_kernel_bench/main.cpp b/examples/cpu_kernel_bench/main.cpp new file mode 100644 index 000000000..a75f106d0 --- /dev/null +++ b/examples/cpu_kernel_bench/main.cpp @@ -0,0 +1,873 @@ +// BACKEND-CPU developer harness. See +// .agents/specs/rpi5-cortex-a76-cpu-optimization.md. This is deliberately an +// op-level tool, not a production API: later reached-loop fixtures plug into +// the same CLI/result schema without putting benchmark machinery in libvllm. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if defined(__linux__) +#include +#include +#include +#include +#include +#endif + +#include "vt/device.h" +#include "vt/dtype.h" +#include "vt/ops.h" +#include "vt/quant.h" +#include "vt/tensor.h" + +namespace { + +using Clock = std::chrono::steady_clock; + +struct Options { + std::string op = "matmul-bt-quant"; + std::string dtype = "q8_0"; + int64_t m = 1; + int64_t n = 3072; + int64_t k = 2048; + int threads = 1; + std::string variant = "auto"; + int warmup = 2; + int iterations = 10; + std::string cache = "hot"; + std::string counters = "auto"; + std::string format = "text"; + uint32_t seed = 0x6a11U; + bool pin = true; +}; + +[[noreturn]] void Fail(const std::string& message) { + throw std::runtime_error(message); +} + +std::string ReadText(const std::string& path) { + std::ifstream in(path); + if (!in) return {}; + std::ostringstream out; + out << in.rdbuf(); + std::string value = out.str(); + while (!value.empty() && (value.back() == '\n' || value.back() == '\r')) value.pop_back(); + return value; +} + +template +T ParseInteger(std::string_view value, std::string_view flag) { + T parsed{}; + const auto result = std::from_chars(value.data(), value.data() + value.size(), parsed); + if (result.ec != std::errc{} || result.ptr != value.data() + value.size()) { + Fail("invalid integer for --" + std::string(flag) + ": " + std::string(value)); + } + return parsed; +} + +void Usage(std::ostream& out) { + out << "vllm-cpu-kernel-bench, PMU-backed CPU operation harness\n\n" + << " --op matmul-bt-quant --dtype q8_0|q4_k|q6_k\n" + << " --m N --n N --k N --threads N --variant auto|portable|repacked|mmla\n" + << " --warmup N --iterations N --cache hot|l2|l3|stream\n" + << " --counters auto|off|generic|a76|all --format text|json\n" + << " --seed N --no-pin\n\n" + << "Cache names are reproducible pressure profiles, not residency claims: l2\n" + << "evicts L1, l3 evicts private L2, and stream exceeds the last-level cache.\n"; +} + +Options ParseArgs(int argc, char** argv) { + Options o; + for (int i = 1; i < argc; ++i) { + std::string arg = argv[i]; + if (arg == "--help" || arg == "-h") { + Usage(std::cout); + std::exit(0); + } + if (arg == "--no-pin") { + o.pin = false; + continue; + } + if (!arg.starts_with("--")) Fail("unexpected positional argument: " + arg); + std::string key; + std::string value; + const size_t eq = arg.find('='); + if (eq != std::string::npos) { + key = arg.substr(2, eq - 2); + value = arg.substr(eq + 1); + } else { + key = arg.substr(2); + if (++i >= argc) Fail("missing value for --" + key); + value = argv[i]; + } + if (key == "op") + o.op = value; + else if (key == "dtype") + o.dtype = value; + else if (key == "m") + o.m = ParseInteger(value, key); + else if (key == "n") + o.n = ParseInteger(value, key); + else if (key == "k") + o.k = ParseInteger(value, key); + else if (key == "threads") + o.threads = ParseInteger(value, key); + else if (key == "variant") + o.variant = value; + else if (key == "warmup") + o.warmup = ParseInteger(value, key); + else if (key == "iterations") + o.iterations = ParseInteger(value, key); + else if (key == "cache") + o.cache = value; + else if (key == "counters") + o.counters = value; + else if (key == "format") + o.format = value; + else if (key == "seed") + o.seed = ParseInteger(value, key); + else + Fail("unknown option --" + key); + } + if (o.op != "matmul-bt-quant") Fail("unsupported --op: " + o.op); + if (o.m <= 0 || o.n <= 0 || o.k <= 0) Fail("M, N and K must be positive"); + if (o.threads <= 0 || o.threads > 512) Fail("--threads must be in [1,512]"); + if (o.warmup < 0 || o.iterations <= 0) Fail("warmup must be >=0 and iterations >0"); + const std::array variants = {"auto", "portable", "repacked", "mmla"}; + const std::array caches = {"hot", "l2", "l3", "stream"}; + const std::array counters = {"auto", "off", "generic", "a76", "all"}; + if (std::find(variants.begin(), variants.end(), o.variant) == variants.end()) + Fail("bad --variant"); + if (std::find(caches.begin(), caches.end(), o.cache) == caches.end()) Fail("bad --cache"); + if (std::find(counters.begin(), counters.end(), o.counters) == counters.end()) + Fail("bad --counters"); + if (o.format != "text" && o.format != "json") Fail("--format must be text or json"); + return o; +} + +void SetEnv(const char* name, const std::string& value) { +#if defined(_WIN32) + if (_putenv_s(name, value.c_str()) != 0) Fail(std::string("cannot set ") + name); +#else + if (setenv(name, value.c_str(), 1) != 0) Fail(std::string("cannot set ") + name); +#endif +} + +std::string CompilerIdentity() { +#if defined(__clang__) + return std::string("clang ") + __clang_version__; +#elif defined(__GNUC__) + return std::string("gcc ") + __VERSION__; +#elif defined(_MSC_VER) + return "msvc " + std::to_string(_MSC_VER); +#else + return "unknown"; +#endif +} + +std::string CompileFeatures() { + std::string s; +#if defined(NDEBUG) + s += "release"; +#else + s += "debug"; +#endif +#if defined(__ARM_FEATURE_DOTPROD) + s += ",dotprod"; +#endif +#if defined(__ARM_FEATURE_MATMUL_INT8) + s += ",i8mm"; +#endif +#if defined(__AVX512F__) + s += ",avx512f"; +#elif defined(__AVX2__) + s += ",avx2"; +#endif + return s; +} + +std::string CpuIdentity() { + std::ifstream in("/proc/cpuinfo"); + std::string line; + while (std::getline(in, line)) { + for (const std::string_view key : {"model name", "Hardware"}) { + if (line.starts_with(key)) { + const size_t colon = line.find(':'); + if (colon != std::string::npos) return line.substr(colon + 2); + } + } + } + return "unavailable"; +} + +std::string Affinity() { +#if defined(__linux__) + cpu_set_t mask; + CPU_ZERO(&mask); + if (sched_getaffinity(0, sizeof(mask), &mask) != 0) return "unavailable"; + std::ostringstream out; + bool first = true; + for (int cpu = 0; cpu < CPU_SETSIZE; ++cpu) { + if (!CPU_ISSET(cpu, &mask)) continue; + if (!first) out << ','; + out << cpu; + first = false; + } + return out.str(); +#else + return "unavailable"; +#endif +} + +void PinToFirstCpus(int count) { +#if defined(__linux__) + cpu_set_t allowed; + CPU_ZERO(&allowed); + if (sched_getaffinity(0, sizeof(allowed), &allowed) != 0) Fail("sched_getaffinity failed"); + cpu_set_t chosen; + CPU_ZERO(&chosen); + int found = 0; + for (int cpu = 0; cpu < CPU_SETSIZE && found < count; ++cpu) { + if (CPU_ISSET(cpu, &allowed)) { + CPU_SET(cpu, &chosen); + ++found; + } + } + if (found != count) Fail("affinity contains fewer CPUs than --threads"); + if (sched_setaffinity(0, sizeof(chosen), &chosen) != 0) Fail("sched_setaffinity failed"); +#else + (void)count; +#endif +} + +std::optional ReadNumber(const std::string& path, double scale) { + const std::string text = ReadText(path); + if (text.empty()) return std::nullopt; + try { + return std::stod(text) * scale; + } catch (...) { + return std::nullopt; + } +} + +std::string ThrottlingStatus() { +#if defined(__linux__) + FILE* pipe = popen("vcgencmd get_throttled 2>/dev/null", "r"); + if (pipe == nullptr) return "unavailable"; + std::array buffer{}; + const char* read = std::fgets(buffer.data(), static_cast(buffer.size()), pipe); + const int status = pclose(pipe); + if (read == nullptr || status != 0) return "unavailable"; + std::string value(buffer.data()); + while (!value.empty() && (value.back() == '\n' || value.back() == '\r')) value.pop_back(); + return value; +#else + return "unavailable"; +#endif +} + +struct EventSpec { + std::string name; + uint32_t type = 0; + uint64_t config = 0; +}; + +struct CounterValue { + std::string name; + double value = 0.0; +}; + +struct CounterPass { + std::string name; + std::string status = "unsupported"; + std::string error; + double running_ratio = 0.0; + std::vector values; +}; + +#if defined(__linux__) +class PerfGroup { + public: + PerfGroup(std::string name, const std::vector& specs) : name_(std::move(name)) { + if (specs.empty()) error_ = "no matching PMU events"; + for (const EventSpec& spec : specs) Open(spec); + } + ~PerfGroup() { + for (int fd : fds_) close(fd); + } + PerfGroup(const PerfGroup&) = delete; + PerfGroup& operator=(const PerfGroup&) = delete; + PerfGroup(PerfGroup&& other) noexcept { *this = std::move(other); } + PerfGroup& operator=(PerfGroup&& other) noexcept { + if (this == &other) return *this; + for (int fd : fds_) close(fd); + name_ = std::move(other.name_); + error_ = std::move(other.error_); + fds_ = std::move(other.fds_); + names_ = std::move(other.names_); + ids_ = std::move(other.ids_); + other.fds_.clear(); + return *this; + } + bool Ready() const { return !fds_.empty(); } + const std::string& Name() const { return name_; } + const std::string& Error() const { return error_; } + void Start() { + if (!Ready()) return; + if (ioctl(fds_[0], PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP) != 0 || + ioctl(fds_[0], PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) != 0) { + error_ = std::strerror(errno); + } + } + CounterPass Stop() { + CounterPass pass; + pass.name = name_; + pass.error = error_; + if (!Ready()) return pass; + if (ioctl(fds_[0], PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP) != 0) { + pass.error = std::strerror(errno); + return pass; + } + std::vector data(3 + 2 * fds_.size()); + const ssize_t wanted = static_cast(data.size() * sizeof(uint64_t)); + if (read(fds_[0], data.data(), static_cast(wanted)) != wanted) { + pass.error = std::strerror(errno); + return pass; + } + const uint64_t nr = data[0]; + const uint64_t enabled = data[1]; + const uint64_t running = data[2]; + if (nr != fds_.size() || enabled == 0 || running == 0) { + pass.status = "unscheduled"; + pass.error = "zero PMU running time or incomplete group read"; + return pass; + } + pass.running_ratio = static_cast(running) / static_cast(enabled); + const double scale = static_cast(enabled) / static_cast(running); + for (size_t i = 0; i < static_cast(nr); ++i) { + const uint64_t raw = data[3 + 2 * i]; + const uint64_t id = data[4 + 2 * i]; + const auto it = std::find(ids_.begin(), ids_.end(), id); + if (it == ids_.end()) continue; + const size_t index = static_cast(it - ids_.begin()); + pass.values.push_back({names_[index], static_cast(raw) * scale}); + } + if (!error_.empty()) + pass.status = "partial"; + else + pass.status = pass.running_ratio < 0.999 ? "multiplexed" : "ok"; + return pass; + } + + private: + void Open(const EventSpec& spec) { + perf_event_attr attr{}; + attr.size = sizeof(attr); + attr.type = spec.type; + attr.config = spec.config; + attr.disabled = fds_.empty() ? 1U : 0U; + attr.inherit = 1U; + attr.inherit_stat = 1U; + attr.exclude_kernel = 1U; + attr.exclude_hv = 1U; + attr.read_format = PERF_FORMAT_GROUP | PERF_FORMAT_ID | PERF_FORMAT_TOTAL_TIME_ENABLED | + PERF_FORMAT_TOTAL_TIME_RUNNING; + const int leader = fds_.empty() ? -1 : fds_[0]; + const int fd = static_cast(syscall(__NR_perf_event_open, &attr, 0, -1, leader, 0)); + if (fd < 0) { + if (!error_.empty()) error_ += "; "; + error_ += spec.name + ": " + std::strerror(errno); + return; + } + uint64_t id = 0; + if (ioctl(fd, PERF_EVENT_IOC_ID, &id) != 0) { + if (error_.empty()) error_ = spec.name + ": cannot read event id"; + close(fd); + return; + } + fds_.push_back(fd); + names_.push_back(spec.name); + ids_.push_back(id); + } + std::string name_; + std::string error_; + std::vector fds_; + std::vector names_; + std::vector ids_; +}; +#else +class PerfGroup { + public: + PerfGroup(std::string name, const std::vector&) : name_(std::move(name)) {} + bool Ready() const { return false; } + const std::string& Name() const { return name_; } + const std::string& Error() const { return error_; } + void Start() {} + CounterPass Stop() { return CounterPass{name_, "unsupported", error_, 0.0, {}}; } + + private: + std::string name_; + std::string error_ = "perf_event_open is Linux-only"; +}; +#endif + +std::optional ParsePerfEvent(std::string_view descriptor) { + const size_t at = descriptor.find("event="); + if (at == std::string_view::npos) return std::nullopt; + const size_t start = at + 6; + size_t end = start; + while (end < descriptor.size() && descriptor[end] != ',' && descriptor[end] != '\n') ++end; + uint64_t value = 0; + std::string text(descriptor.substr(start, end - start)); + char* parsed_end = nullptr; + value = std::strtoull(text.c_str(), &parsed_end, 0); + if (parsed_end == text.c_str() || *parsed_end != '\0') return std::nullopt; + return value; +} + +std::optional A76Event(const std::string& name) { + const std::string root = "/sys/bus/event_source/devices/armv8_cortex_a76/"; + const std::string type_text = ReadText(root + "type"); + const std::string descriptor = ReadText(root + "events/" + name); + if (type_text.empty() || descriptor.empty()) return std::nullopt; + const auto config = ParsePerfEvent(descriptor); + if (!config) return std::nullopt; + return EventSpec{name, ParseInteger(type_text, "pmu-type"), *config}; +} + +std::vector MakeCounterGroups(const std::string& selection) { + std::vector groups; + if (selection == "off") return groups; +#if defined(__linux__) + const bool generic = selection == "auto" || selection == "generic" || selection == "all"; + const bool a76 = selection == "auto" || selection == "a76" || selection == "all"; + if (generic) { + groups.emplace_back( + "generic-core", + std::vector{ + {"cycles", PERF_TYPE_HARDWARE, PERF_COUNT_HW_CPU_CYCLES}, + {"instructions", PERF_TYPE_HARDWARE, PERF_COUNT_HW_INSTRUCTIONS}, + {"frontend_stalls", PERF_TYPE_HARDWARE, PERF_COUNT_HW_STALLED_CYCLES_FRONTEND}, + {"backend_stalls", PERF_TYPE_HARDWARE, PERF_COUNT_HW_STALLED_CYCLES_BACKEND}}); + groups.emplace_back( + "generic-memory-branch", + std::vector{{"cycles", PERF_TYPE_HARDWARE, PERF_COUNT_HW_CPU_CYCLES}, + {"instructions", PERF_TYPE_HARDWARE, PERF_COUNT_HW_INSTRUCTIONS}, + {"cache_misses", PERF_TYPE_HARDWARE, PERF_COUNT_HW_CACHE_MISSES}, + {"branch_misses", PERF_TYPE_HARDWARE, PERF_COUNT_HW_BRANCH_MISSES}}); + } + if (a76) { + const size_t before = groups.size(); + for (const auto& names : std::array, 4>{ + {{{"cpu_cycles", "inst_retired", "stall_frontend", "stall_backend"}}, + {{"cpu_cycles", "inst_retired", "l1d_cache_refill", "l2d_cache_refill"}}, + {{"cpu_cycles", "inst_retired", "ll_cache_rd", "ll_cache_miss_rd"}}, + {{"cpu_cycles", "inst_retired", "dtlb_walk", "br_mis_pred"}}}}) { + std::vector specs; + for (const char* name : names) { + if (auto event = A76Event(name)) specs.push_back(*event); + } + if (specs.size() == names.size()) groups.emplace_back(std::string("a76-") + names[2], specs); + } + if (groups.size() == before) groups.emplace_back("a76-pmu", std::vector{}); + } +#else + (void)selection; +#endif + return groups; +} + +vt::DType ParseDType(const std::string& name) { + if (name == "q8_0") return vt::DType::kQ8_0; + if (name == "q4_k") return vt::DType::kQ4_K; + if (name == "q6_k") return vt::DType::kQ6_K; + Fail("unsupported --dtype: " + name); +} + +int64_t BlockElements(vt::DType dtype) { + return dtype == vt::DType::kQ8_0 ? 32 : 256; +} + +std::vector RandomBlocks(vt::DType dtype, int64_t nblocks, uint32_t seed) { + const int64_t be = BlockElements(dtype); + const size_t block_bytes = vt::RowSizeBytes(dtype, be); + std::mt19937 rng(seed); + std::vector bytes(static_cast(nblocks) * block_bytes); + for (uint8_t& byte : bytes) byte = static_cast(rng() & 0xffU); + const int d_offset = dtype == vt::DType::kQ6_K ? 208 : 0; + const int dmin_offset = dtype == vt::DType::kQ4_K ? 2 : -1; + for (int64_t i = 0; i < nblocks; ++i) { + uint8_t* block = bytes.data() + static_cast(i) * block_bytes; + const float jitter = 1.0F + 0.05F * static_cast(i % 7); + auto put = [&](int offset, float value) { + const uint16_t half = vt::F32ToF16(value); + std::memcpy(block + offset, &half, sizeof(half)); + }; + put(d_offset, 0.0125F * jitter); + if (dmin_offset >= 0) put(dmin_offset, 0.0075F * jitter); + } + return bytes; +} + +class QuantMatmulFixture { + public: + explicit QuantMatmulFixture(const Options& o) + : dtype_(ParseDType(o.dtype)), q_({vt::DeviceType::kCPU, 0}, nullptr) { + const int64_t be = BlockElements(dtype_); + if (o.k % be != 0) Fail("K must be divisible by the quant block size"); + if (o.m > std::numeric_limits::max() / o.k || + o.n > std::numeric_limits::max() / o.k || + o.m > std::numeric_limits::max() / o.n) + Fail("shape overflow"); + weights_ = RandomBlocks(dtype_, o.n * (o.k / be), o.seed); + std::mt19937 rng(o.seed ^ 0x51f15eU); + activations_.resize(static_cast(o.m * o.k)); + for (float& x : activations_) { + const int centered = static_cast(rng() % 2001U) - 1000; + x = 0.1F + 0.001F * static_cast(centered); + } + output_.resize(static_cast(o.m * o.n)); + if (o.variant == "repacked") { + if (dtype_ != vt::DType::kQ8_0 || !vt::cpu::QuantRepackActive()) { + Fail("repacked requires q8_0 and an active repack provider"); + } + vt::cpu::QuantRepackWeight(dtype_, weights_.data(), o.n, o.k); + repacked_ = true; + } + if (o.variant == "mmla" && !vt::cpu::QuantMmlaActive()) Fail("mmla is unavailable"); + at_ = vt::Tensor::Contiguous(activations_.data(), vt::DType::kF32, q_.device, {o.m, o.k}); + bt_ = vt::Tensor::Contiguous(weights_.data(), vt::DType::kF32, q_.device, {o.n, o.k}); + bt_.dtype = dtype_; + bt_.repacked = repacked_; + ot_ = vt::Tensor::Contiguous(output_.data(), vt::DType::kF32, q_.device, {o.m, o.n}); + } + void Run() { vt::MatmulBTQuant(q_, ot_, at_, bt_); } + uint64_t Checksum() const { + uint64_t hash = 1469598103934665603ULL; + const auto* bytes = reinterpret_cast(output_.data()); + for (size_t i = 0; i < output_.size() * sizeof(float); ++i) { + hash = (hash ^ bytes[i]) * 1099511628211ULL; + } + return hash; + } + uint64_t BytesPerCall() const { + return static_cast(weights_.size()) + + static_cast((activations_.size() + output_.size()) * sizeof(float)); + } + + private: + vt::DType dtype_; + vt::Queue q_; + std::vector weights_; + std::vector activations_; + std::vector output_; + vt::Tensor at_; + vt::Tensor bt_; + vt::Tensor ot_; + bool repacked_ = false; +}; + +size_t CachePressureBytes(const std::string& mode) { + if (mode == "hot") return 0; + if (mode == "l2") return 128U * 1024U; + if (mode == "l3") return 1024U * 1024U; + return 16U * 1024U * 1024U; +} + +void Touch(std::vector& buffer) { + static volatile uint64_t sink = 0; + uint64_t sum = 0; + for (size_t i = 0; i < buffer.size(); i += 64) { + buffer[i] = static_cast(buffer[i] + 1U); + sum += buffer[i]; + } + sink = sink ^ sum; +} + +double TimerOverheadNs() { + double best = std::numeric_limits::infinity(); + for (int i = 0; i < 1000; ++i) { + const auto begin = Clock::now(); + const auto end = Clock::now(); + best = std::min(best, std::chrono::duration(end - begin).count()); + } + return best; +} + +struct Result { + size_t calls_per_sample = 1; + std::vector sample_ns; + double median_ns = 0.0; + double min_ns = 0.0; + double mean_ns = 0.0; + double timer_overhead_ratio = 0.0; + uint64_t checksum = 0; + uint64_t bytes_per_call = 0; + std::vector counters; + int cpu_before = -1; + int cpu_after = -1; + std::optional freq_before_mhz; + std::optional freq_after_mhz; + std::optional temp_before_c; + std::optional temp_after_c; + std::string governor = "unavailable"; + std::string throttle_before = "unavailable"; + std::string throttle_after = "unavailable"; +}; + +int CurrentCpu() { +#if defined(__linux__) + return sched_getcpu(); +#else + return -1; +#endif +} + +Result Benchmark(const Options& o, QuantMatmulFixture& fixture, std::vector& groups) { + Result result; + result.bytes_per_call = fixture.BytesPerCall(); + std::vector pressure(CachePressureBytes(o.cache)); + for (int i = 0; i < o.warmup; ++i) fixture.Run(); + const double overhead = TimerOverheadNs(); + if (o.cache == "hot") { + for (;;) { + const auto begin = Clock::now(); + for (size_t i = 0; i < result.calls_per_sample; ++i) fixture.Run(); + const auto end = Clock::now(); + const double ns = std::chrono::duration(end - begin).count(); + if (ns >= std::max(2.0e6, 1000.0 * overhead) || result.calls_per_sample >= (1U << 20)) break; + result.calls_per_sample *= 2; + } + } + result.cpu_before = CurrentCpu(); + if (result.cpu_before >= 0) { + const std::string root = + "/sys/devices/system/cpu/cpu" + std::to_string(result.cpu_before) + "/cpufreq/"; + result.freq_before_mhz = ReadNumber(root + "scaling_cur_freq", 0.001); + const std::string governor = ReadText(root + "scaling_governor"); + if (!governor.empty()) result.governor = governor; + } + result.temp_before_c = ReadNumber("/sys/class/thermal/thermal_zone0/temp", 0.001); + result.throttle_before = ThrottlingStatus(); + for (int sample = 0; sample < o.iterations; ++sample) { + Touch(pressure); + const auto begin = Clock::now(); + for (size_t call = 0; call < result.calls_per_sample; ++call) fixture.Run(); + const auto end = Clock::now(); + const double ns = std::chrono::duration(end - begin).count() / + static_cast(result.calls_per_sample); + result.sample_ns.push_back(ns); + } + result.cpu_after = CurrentCpu(); + if (result.cpu_after >= 0) { + const std::string root = + "/sys/devices/system/cpu/cpu" + std::to_string(result.cpu_after) + "/cpufreq/"; + result.freq_after_mhz = ReadNumber(root + "scaling_cur_freq", 0.001); + } + result.temp_after_c = ReadNumber("/sys/class/thermal/thermal_zone0/temp", 0.001); + result.throttle_after = ThrottlingStatus(); + std::vector sorted = result.sample_ns; + std::sort(sorted.begin(), sorted.end()); + result.min_ns = sorted.front(); + result.median_ns = sorted[sorted.size() / 2]; + double total = 0.0; + for (double ns : result.sample_ns) total += ns; + result.mean_ns = total / static_cast(result.sample_ns.size()); + result.timer_overhead_ratio = + overhead / (result.median_ns * static_cast(result.calls_per_sample)); + result.checksum = fixture.Checksum(); + for (PerfGroup& group : groups) { + Touch(pressure); + group.Start(); + for (size_t call = 0; call < result.calls_per_sample; ++call) fixture.Run(); + CounterPass pass = group.Stop(); + if (fixture.Checksum() != result.checksum) Fail("output checksum changed during PMU pass"); + for (CounterValue& value : pass.values) + value.value /= static_cast(result.calls_per_sample); + result.counters.push_back(std::move(pass)); + } + return result; +} + +void JsonString(std::ostream& out, std::string_view value) { + out << '"'; + for (const char c : value) { + if (c == '"' || c == '\\') + out << '\\' << c; + else if (c == '\n') + out << "\\n"; + else + out << c; + } + out << '"'; +} + +void JsonOptional(std::ostream& out, const std::optional& value) { + if (value) + out << *value; + else + out << "null"; +} + +std::optional Counter(const CounterPass& pass, std::string_view name) { + for (const CounterValue& value : pass.values) { + if (value.name == name) return value.value; + } + return std::nullopt; +} + +void PrintJson(const Options& o, const Result& r) { + const double seconds = r.median_ns * 1e-9; + const double gflops = 2.0 * static_cast(o.m) * static_cast(o.n) * + static_cast(o.k) / seconds / 1e9; + const double bandwidth = static_cast(r.bytes_per_call) / seconds / 1e9; + const bool timer_valid = r.timer_overhead_ratio < 0.001; + const bool migration_valid = r.cpu_before < 0 || r.cpu_after < 0 || r.cpu_before == r.cpu_after; + std::cout << std::setprecision(10) << '{'; + std::cout << "\"schema\":\"vllm-cpu-kernel-bench/v1\",\"scope\":\"vt-op\",\"op\":"; + JsonString(std::cout, o.op); + std::cout << ",\"dtype\":"; + JsonString(std::cout, o.dtype); + std::cout << ",\"shape\":{\"m\":" << o.m << ",\"n\":" << o.n << ",\"k\":" << o.k << '}'; + std::cout << ",\"threads\":" << o.threads << ",\"variant\":"; + JsonString(std::cout, o.variant); + std::cout << ",\"cache\":"; + JsonString(std::cout, o.cache); + std::cout << ",\"timing\":{\"samples\":" << o.iterations + << ",\"calls_per_sample\":" << r.calls_per_sample << ",\"min_ns\":" << r.min_ns + << ",\"median_ns\":" << r.median_ns << ",\"mean_ns\":" << r.mean_ns + << ",\"timer_overhead_ratio\":" << r.timer_overhead_ratio + << ",\"valid\":" << (timer_valid ? "true" : "false") << '}'; + std::cout << ",\"work\":{\"bytes_per_call\":" << r.bytes_per_call + << ",\"effective_gb_s\":" << bandwidth << ",\"gflop_s\":" << gflops << '}'; + std::ostringstream checksum; + checksum << "0x" << std::hex << std::setw(16) << std::setfill('0') << r.checksum; + std::cout << ",\"checksum\":"; + JsonString(std::cout, checksum.str()); + std::cout << ",\"metadata\":{\"compiler\":"; + JsonString(std::cout, CompilerIdentity()); + std::cout << ",\"compile_features\":"; + JsonString(std::cout, CompileFeatures()); + std::cout << ",\"mmla_active\":" << (vt::cpu::QuantMmlaActive() ? "true" : "false") + << ",\"repack_active\":" << (vt::cpu::QuantRepackActive() ? "true" : "false"); + std::cout << ",\"cpu\":"; + JsonString(std::cout, CpuIdentity()); + std::cout << ",\"affinity\":"; + JsonString(std::cout, Affinity()); + std::cout << ",\"cpu_before\":" << r.cpu_before << ",\"cpu_after\":" << r.cpu_after; + std::cout << ",\"frequency_before_mhz\":"; + JsonOptional(std::cout, r.freq_before_mhz); + std::cout << ",\"frequency_after_mhz\":"; + JsonOptional(std::cout, r.freq_after_mhz); + std::cout << ",\"temperature_before_c\":"; + JsonOptional(std::cout, r.temp_before_c); + std::cout << ",\"temperature_after_c\":"; + JsonOptional(std::cout, r.temp_after_c); + std::cout << ",\"governor\":"; + JsonString(std::cout, r.governor); + std::cout << ",\"throttle_before\":"; + JsonString(std::cout, r.throttle_before); + std::cout << ",\"throttle_after\":"; + JsonString(std::cout, r.throttle_after); + std::cout << ",\"migration_valid\":" << (migration_valid ? "true" : "false") << '}'; + std::cout << ",\"counter_passes\":["; + for (size_t i = 0; i < r.counters.size(); ++i) { + const CounterPass& pass = r.counters[i]; + if (i != 0) std::cout << ','; + std::cout << "{\"name\":"; + JsonString(std::cout, pass.name); + std::cout << ",\"status\":"; + JsonString(std::cout, pass.status); + std::cout << ",\"error\":"; + JsonString(std::cout, pass.error); + std::cout << ",\"running_ratio\":" << pass.running_ratio << ",\"ipc\":"; + const auto cycles = Counter(pass, pass.name.starts_with("a76-") ? "cpu_cycles" : "cycles"); + const auto instructions = + Counter(pass, pass.name.starts_with("a76-") ? "inst_retired" : "instructions"); + if (cycles && instructions && *cycles > 0.0) + std::cout << *instructions / *cycles; + else + std::cout << "null"; + std::cout << ",\"events\":{"; + for (size_t j = 0; j < pass.values.size(); ++j) { + if (j != 0) std::cout << ','; + JsonString(std::cout, pass.values[j].name); + std::cout << ':' << pass.values[j].value; + } + std::cout << "}}"; + } + std::cout << "]}\n"; +} + +void PrintText(const Options& o, const Result& r) { + const double seconds = r.median_ns * 1e-9; + const double gflops = 2.0 * static_cast(o.m) * static_cast(o.n) * + static_cast(o.k) / seconds / 1e9; + std::cout << "vllm-cpu-kernel-bench/v1 " << o.op << ' ' << o.dtype << " M=" << o.m << " N=" << o.n + << " K=" << o.k << " threads=" << o.threads << " variant=" << o.variant + << " cache=" << o.cache << '\n'; + std::cout << std::fixed << std::setprecision(3) << "median " << r.median_ns / 1e6 << " ms, min " + << r.min_ns / 1e6 << " ms, " << gflops << " GFLOP/s, timer ratio " + << 100.0 * r.timer_overhead_ratio << "%\n"; + std::cout << "affinity " << Affinity() << ", CPU " << r.cpu_before << " -> " << r.cpu_after + << ", checksum 0x" << std::hex << r.checksum << std::dec << '\n'; + for (const CounterPass& pass : r.counters) { + std::cout << pass.name << ": " << pass.status << " running=" << 100.0 * pass.running_ratio + << '%'; + if (!pass.error.empty()) std::cout << " (" << pass.error << ')'; + for (const CounterValue& value : pass.values) { + std::cout << " " << value.name << '=' << std::setprecision(0) << value.value; + } + const auto cycles = Counter(pass, pass.name.starts_with("a76-") ? "cpu_cycles" : "cycles"); + const auto instructions = + Counter(pass, pass.name.starts_with("a76-") ? "inst_retired" : "instructions"); + if (cycles && instructions && *cycles > 0.0) { + std::cout << " IPC=" << std::setprecision(3) << *instructions / *cycles; + } + std::cout << '\n'; + } +} + +} // namespace + +int main(int argc, char** argv) { + try { + Options options = ParseArgs(argc, argv); + SetEnv("VLLM_CPP_CPU_THREADS", std::to_string(options.threads)); + if (options.variant == "portable") { + SetEnv("VT_CPU_QUANT_MMLA", "0"); + SetEnv("VT_CPU_QUANT_REPACK", "0"); + } + if (options.pin) PinToFirstCpus(options.threads); + QuantMatmulFixture fixture(options); + // Open before the warmup creates persistent workers. inherit=1 then makes + // each pass process-wide rather than silently counting only worker zero. + std::vector groups = MakeCounterGroups(options.counters); + Result result = Benchmark(options, fixture, groups); + if (options.format == "json") + PrintJson(options, result); + else + PrintText(options, result); + return 0; + } catch (const std::exception& error) { + std::cerr << "vllm-cpu-kernel-bench: " << error.what() << '\n'; + return 2; + } +} diff --git a/tests/scripts/test_cpu_kernel_bench.py b/tests/scripts/test_cpu_kernel_bench.py new file mode 100644 index 000000000..e3e2c5175 --- /dev/null +++ b/tests/scripts/test_cpu_kernel_bench.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Black-box contract for the BACKEND-CPU PMU benchmark CLI.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import unittest +from pathlib import Path + +if len(sys.argv) != 2: + raise RuntimeError("expected the benchmark executable path") +BENCH_BINARY = Path(sys.argv.pop()) + + +class CpuKernelBenchTests(unittest.TestCase): + def run_bench(self, *args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [str(BENCH_BINARY), *args], + check=False, + text=True, + capture_output=True, + ) + + def test_json_schema_and_deterministic_checksum(self) -> None: + args = ( + "--dtype", "q4_k", "--m", "1", "--n", "8", "--k", "256", + "--warmup", "1", "--iterations", "2", "--counters", "off", + "--format", "json", + ) + first = self.run_bench(*args) + second = self.run_bench(*args) + self.assertEqual(first.returncode, 0, first.stderr) + self.assertEqual(second.returncode, 0, second.stderr) + got = json.loads(first.stdout) + again = json.loads(second.stdout) + self.assertEqual(got["schema"], "vllm-cpu-kernel-bench/v1") + self.assertEqual(got["scope"], "vt-op") + self.assertEqual(got["shape"], {"m": 1, "n": 8, "k": 256}) + self.assertTrue(got["timing"]["valid"]) + self.assertGreater(got["timing"]["calls_per_sample"], 1) + self.assertEqual(got["checksum"], again["checksum"]) + self.assertEqual(got["counter_passes"], []) + self.assertIn("compiler", got["metadata"]) + self.assertIn("throttle_before", got["metadata"]) + + def test_invalid_dtype_fails_loudly(self) -> None: + got = self.run_bench("--dtype", "invented") + self.assertEqual(got.returncode, 2) + self.assertIn("unsupported --dtype", got.stderr) + + def test_counter_unavailability_is_structured(self) -> None: + got = self.run_bench( + "--dtype", "q8_0", "--m", "1", "--n", "8", "--k", "256", + "--warmup", "0", "--iterations", "1", "--counters", "a76", + "--format", "json", + ) + self.assertEqual(got.returncode, 0, got.stderr) + passes = json.loads(got.stdout)["counter_passes"] + self.assertGreaterEqual(len(passes), 1) + self.assertIn(passes[0]["status"], {"ok", "partial", "multiplexed", "unsupported"}) + + +if __name__ == "__main__": + unittest.main() From 16c848326b31c46875a4cff1f35d7d68a5ba08c2 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 17:46:35 +0100 Subject: [PATCH 4/7] Checkpoint Raspberry Pi 5 CPU baseline Build ARM64 artifacts under buildx/QEMU, verify the exported binaries and exact Qwen3.5-2B tokens on Cortex-A76, and record the PMU-ranked portable baseline that selects Q8 SDOT as the first assembly candidate. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 2 +- .agents/backend-matrix.md | 2 +- .agents/feature-matrix.md | 2 +- .../specs/rpi5-cortex-a76-cpu-optimization.md | 64 ++++++++++++++++--- .agents/state.md | 31 +++++++++ .dockerignore | 5 ++ docker/Dockerfile.arm64 | 10 +++ docs/BENCHMARKS.md | 21 +++--- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- examples/CMakeLists.txt | 2 - 11 files changed, 120 insertions(+), 23 deletions(-) create mode 100644 .dockerignore create mode 100644 docker/Dockerfile.arm64 diff --git a/.agents/NOW.md b/.agents/NOW.md index 1f1dd9c84..9567ff8bf 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| RPi5 A76 CPU | **R1 HARNESS CPU-GREEN PR #65**; Pi/model evidence pending | Run harness on Pi, then Qwen3.5-2B Q8 bring-up | +| RPi5 A76 CPU | **R3 BASELINE GREEN:** QEMU-built artifact, 16/16 x86-token exact, PMU trace | Portable Q8 dot is 20.1% of user cycles; prove C++ SDOT/compiler gap, then A76 assembly A/B | | MXFP4 parity | **TERMINAL:** c1 1.020x pass; c2-c8 0.962-0.969 GPU-intrinsic; final cap/glue levers exhausted | Record; branch unmerged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 7926b7310..f137ea0cf 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -223,7 +223,7 @@ on every listed target. | `BACKEND-ABI-VT` | Backend registry, device/queue, capture, drop-in adapter ABI | platform contract `vllm/platforms/interface.py:67-229`; stable stream `csrc/libtorch_stable/torch_utils.h:76-82`; raw Marlin boundary `csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu:344-356` | queue ID [device.h:9](../include/vt/device.h#L9), explicit resources [backend.h:53](../include/vt/backend.h#L53), scalar/layout ABI [ops.h:13](../include/vt/ops.h#L13), CUDA workspace/raw probe [cuda_dropin.cu:104](../src/vt/cuda/cuda_dropin.cu#L104) | [ported ABI tests](../tests/vt/test_dropin_abi.cpp#L86); CPU 94/94 + repair-focused 1/1; GCC13 repair at [test_dropin_abi.cpp:245](../tests/vt/test_dropin_abi.cpp#L245); exact sm_121a all-target build 100%, focused CUDA/ABI CTest 2/2, sanitizer 9/9 + 196/196 with 0 errors/leaks, 35B/27B gates 2/2 (`1141b79`, evidence manifest `4adbe952…601`) | [drop-in ABI](specs/dropin-kernel-abi.md) | `ANCHOR-BACKFILL` - sm_80/sm_90a cross-build and unchanged-trace/model A/B-memory proof remain; common scalar forwarder and legacy backend-shim migration remain; no production family migrated | CLAIM-BACKEND-ABI-W0-GPU-1 | | `BACKEND-ATTN-REGISTRY` | Attention-backend registry + platform-driven priority selection (extensibility **item 4** — the second portability seam, now REALIZED). Backends SELF-REGISTER per `(DeviceType, name)`; `Platform::get_attn_backend_priority()` advertises a capability-ordered name list; the selector returns the first REGISTERED name. Selection is DATA (register + priority slot), not an inline code edit. The concrete attention KERNEL stays at the vt:: op-table (`vt::PagedAttention`→`GetOp`), already device-additive | `vllm/v1/attention/backends/registry.py` (self-registration), `vllm/platforms/cuda.py:361-470` (`get_valid_backends`/`get_attn_backend_cls`) + `:84-166` (`_get_backend_priorities`, non-MLA major-10 vs else), `cpu.py:75-87` (CPU_ATTN) @ pin `e24d1b24` | registry+selector [registry.h:44](../include/vllm/v1/attention/registry.h#L44), [registry.cpp:60](../src/vllm/v1/attention/registry.cpp#L60); priority [interface.h:92](../include/vllm/platforms/interface.h#L92), CUDA major-10/else [cuda.cpp:44](../src/vllm/platforms/cuda.cpp#L44), CPU [cpu.cpp:29](../src/vllm/platforms/cpu.cpp#L29); FLASH_ATTN self-reg [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), GDN [gdn_attn.cpp:153](../src/vllm/v1/attention/backends/gdn_attn.cpp#L153) | [registry tests](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L61) (self-register + Make/throw + CUDA/CPU priority order + first-registered walk + override; 8 cases/25 assertions) + [platform priority](../tests/vllm/platforms/test_platform.cpp#L53); clean CPU `-Werror` + full CPU CTest green; behavior-preserving ⇒ same FA2 attention selected (**DGX-CONFIRMED @ `2c732e7`: 27B 235/235 + 35B 315/315 token-exact, FA2 sm_121a, memcheck 0/315**) | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) (item 4) | `ANCHOR-BACKFILL` — item-4 seam LANDED CPU + DGX-CONFIRMED; FLASHINFER/TRITON_ATTN/FLEX/TURBOQUANT are named in the priority (fidelity) but unimplemented ⇒ walk falls through to FLASH_ATTN (the behavior-preserving outcome); MLA-branch priorities deferred until an MLA model ports | CLAIM-ATTN-REGISTRY-1 | | `BACKEND-PLATFORM` | Platform capability/memory-model seam (`is_cuda`/`is_cpu`, `is_unified_memory`, `has_device_capability`, `supported_dtypes`, `residency_policy`, `supports_graph_capture`; `get_attn_backend_priority` REALIZED by `BACKEND-ATTN-REGISTRY` item 4; **+ the S3 capability predicates `supports_fp8`/`cutlass_fp4_supported`/`opaque_attention_op`/`is_integrated_gpu`/`support_static_graph_mode`/`is_device_capability_family`, mirroring `interface.py:914,933,977,1058,441-476` + `nvfp4_utils.py:56`, added by `CLAIM-BACKEND-SEAM-S3-1` and consumed by the fp4/fp8/graph gates in `qwen3_5.cpp`**); self-registered per `DeviceType` via `CurrentPlatform()` | `vllm/platforms/interface.py:134-229` (`class Platform`), `:409-439` (get/has_device_capability), `:181-187` (supported_dtypes) @ pin `e24d1b24` | interface [interface.h:56](../include/vllm/platforms/interface.h#L56), registry/CurrentPlatform [platform.cpp:57](../src/vllm/platforms/platform.cpp#L57), CPU [cpu.cpp:11](../src/vllm/platforms/cpu.cpp#L11), CUDA [cuda.cpp:14](../src/vllm/platforms/cuda.cpp#L14); composes `vt::Backend` [backend.h:22](../include/vt/backend.h#L22) | [platform tests](../tests/vllm/platforms/test_platform.cpp#L28) (registration + CPU capabilities + has_device_capability), clean CPU `-Werror` + full CPU CTest; 7 memory-model/residency sites migrated (`runner.cpp`, `model_registry.cpp`, `qwen3_5.cpp`) — **per-tensor sites key on `GetPlatform(.device.type).is_cuda()` (the OBJECT's device), NOT the process-global `CurrentPlatform()`, which is accelerator-first and would mis-route a CPU queue/tensor on a GPU box (regression fixed 2026-07-18; red DGX CPU tests `test_platform`/`test_qwen27_dense_forward`); `CurrentPlatform()` is reserved for process-level "which accelerator" questions only**. DGX behavior-preserving model gates 27B 235/235 + 35B 315/315 CONFIRMED @ (CUDA -Werror-clean after a GCC13 dangling-pointer FP fix). **Item-2 residency_policy() CONSUMPTION LANDED (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`):** the host-free / load-stream / DevicePool-cap decisions in `qwen3_5.cpp` now READ `GetPlatform(.device.type).residency_policy()` (per-device) instead of an inline `device.type`/env gate — host-free `qwen3_5.cpp:BuildMoeMarlinResident` via `ShouldReleaseHostWeights`, load-stream `Qwen3_5Model::PrepareMarlinResident` via `ShouldInterleaveLoadStream` (both `interface.h`), DevicePool soft cap via `residency_policy().device_pool_cap_bytes` (`DBuf`/`DevicePool`). `CudaPlatform::residency_policy().release_host_weights_after_upload` flipped **false→true** (now CONSUMED ⇒ reproduces today's GB10 host-free-after-Marlin-build EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH gate; `VT_MOE_HOST_FREE`/`VT_MOE_LOADSTREAM` env stay as overrides. Consumption unit-tested `test_platform.cpp` (7 cases/43 assertions). A new (discrete) GPU sets `residency_policy()` values ⇒ ZERO model edit. **DGX gate PASSED @ `62fc0e0` (production flags CUTLASS sm120a + FA2 sm_121a + Triton AOT, one flock): clean CUDA `-Werror` 0 warn, 27B 235/235 + 35B 315/315 token-exact, 35B VmHWM ≈ 4.0 GiB (load-stream win preserved), memcheck 0 errors.** | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) | `ANCHOR-BACKFILL` — item-1 extraction DONE + DGX-CONFIRMED; item-2 residency consumption LANDED + DGX-PROVEN (`CLAIM-BACKEND-PLATFORM-2`); kernel-shape dispatch branches deliberately left for items 4/5 | CLAIM-BACKEND-PLATFORM-1 / CLAIM-BACKEND-PLATFORM-2 | -| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06 (PR #65):** R0 pins the exact Q8_K_XL hash, four-core A76 DotProd/no-i8mm target, recursive gates and llama.cpp floor. R1 [PMU harness](../examples/cpu_kernel_bench/main.cpp) is x86 CPU-gated: parameterized quant GEMM, deterministic checksum, calibrated batched timing, 1/2/4-core affinity, cache-pressure profiles, JSON, generic + A76 raw grouped counters with IPC/multiplex/unsupported status, frequency/temperature/governor/throttle metadata; [black-box CLI test](../tests/scripts/test_cpu_kernel_bench.py). Pi execution, correctness, baseline and every Pi speed number remain `PENDING`; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | +| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06 (PR #65):** R0 pins the exact Q8_K_XL hash, four-core A76 DotProd/no-i8mm target, recursive gates and llama.cpp floor. R1 [PMU harness](../examples/cpu_kernel_bench/main.cpp) is CPU-gated. **R2-R3 GREEN:** QEMU-built AArch64 artifacts pass smoke and execute on the Pi; Qwen3.5-2B is exact 16/16 against the x86 token golden, four operation fixtures retain exact checksums, and unthrottled PMU baselines are captured. A zero-loss model trace ranks portable Q8 dot at 20.10% of user cycles (BF16 GEMM 57.76%); the Pi has DotProd but no i8mm, so C++ SDOT and AAPCS64 assembly A/B are the active next gate. No optimized or llama.cpp-parity result is claimed; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | | `BACKEND-CPU-ZEN` | AMD Zen CPU with ZenDNN/zentorch dispatch and weight prepack | `platforms/zen_cpu.py:12-32`; detection `platforms/__init__.py:153-192`; `tests/test_zen_cpu_platform_detection.py:8-37` | generic CPU backend only; no Zen-specific dispatch | - | [CUDA inventory](specs/cuda-architecture-inventory.md); leaf spike required | `INVENTORIED` | - | | `BACKEND-ROCM` | AMD ROCm/HIP | `platforms/__init__.py:110-128`, `platforms/rocm.py:43-125`, ROCm ops rooted at `csrc/rocm/moe_q_gemm_rdna3.cu:1` | Lands through the realized seams: a `platforms/rocm.cpp` Platform + a `kROCM` attention TU self-registering via the [attn-registry seam](../include/vllm/v1/attention/registry.h#L44) + a `rocm.cpp` `get_attn_backend_priority()` slot (mirror `rocm.py`'s ROCM_ATTN/AITER ordering); ZERO selector/model/runner edit | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-XPU` | Intel XPU loyal port; gating exploration E4 ([backends.md](backends.md) kernel sourcing: SYCL-vs-Level-Zero call + whether upstream XPU attention contracts translate 1:1; explore when T2 scheduling begins) | `platforms/__init__.py:131-150`, `platforms/xpu.py:103-125` | enum slot only [device.h:11](../include/vt/device.h#L11) | [unavailable-backend test](../tests/vt/test_backend.cpp#L37); **SPIKE-ONLY verdict, MEASURED 2026-07-22: HW-BLOCKED — no Intel GPU on any box we own, and no acquisition path recorded.** Compounding finding: the loyalty target is INCOMPLETE upstream — `vllm/platforms/xpu.py:11-13` imports the EXTERNAL `vllm_xpu_kernels._C`/`._moe_C`/`._xpu_C`, and a search of the pinned tree for `*.sycl`/`*.dp.cpp` returns NOTHING, so there is no in-tree SYCL kernel source to mirror 1:1. What stays doable with zero Intel hardware: the attention-selector/priority POLICY port (pure data, unit-testable), oneAPI DPC++ compile coverage, and SYCL kernels executed on the oneAPI OpenCL CPU device for unit-level numerics. **No e2e or performance gate is proposed, because none can be run.** No XPU backend code exists | [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `SPIKE` | `CLAIM-BACKEND-FANOUT-1` | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index ad078e39e..24340e2a6 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -274,7 +274,7 @@ evidence. |---|---|---|---|---| | `BACKEND-CUDA-SM121` | GB10/sm121a | `PARTIAL` | gate workload built, traced, token/perf gated; full component-family coverage is open | [backend row](backend-matrix.md#cuda-target-rows) | | `BACKEND-CUDA-OTHER` | vLLM sm70/75/80/86/87/89/90/100/101/103/110/120 targets | `ACTIVE` | 9 CUDA arches build-supported (sm80/86/87/89/90a/100a/103a/110/120a, single-arch portable-kernels-only, `-Werror` clean, SASS emitted); sm70/75/101 not build-supported; no non-121a target is runtime-validated | [backend matrix](backend-matrix.md), [CUDA inventory](specs/cuda-architecture-inventory.md) | -| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 (DotProd, no i8mm) R1 PMU harness is x86 CPU-gated under PR #65; Pi execution, model bring-up and all Pi performance evidence remain pending | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md), [harness](../examples/cpu_kernel_bench/main.cpp) | +| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 R2-R3 is green: QEMU-built artifact, exact 16/16 x86-token and fixture parity, PMU baseline and a zero-loss model trace. Portable Q8 dot is 20.10% of user cycles; SDOT C++/assembly optimization and the Pi llama.cpp floor remain open | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md), [harness](../examples/cpu_kernel_bench/main.cpp) | | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) | | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) | | `BACKEND-VULKAN` | Vulkan | `INVENTORIED` | runtime absent | [backend matrix](backend-matrix.md) | diff --git a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md index 8603f671f..8769a3cb9 100644 --- a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md +++ b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md @@ -40,6 +40,28 @@ the Pi is the performance and memory floor. `perf` and LLVM are not initially installed; direct `perf_event_open` is the binding harness path, with `perf`/`llvm-mca` optional cross-checks. +## Build and execution split + +The Pi is an execution target, not a build host. `docker/Dockerfile.arm64` +mirrors the repository-family buildx pattern: Ubuntu 24.04 runs as +`linux/arm64` under QEMU, configures a Release CPU-only build, compiles +`vllm-bench` plus `vllm-cpu-kernel-bench`, executes a small quantized-matmul +smoke, and exports only those two binaries: + +```sh +docker buildx build --builder pf-arm --platform linux/arm64 \ + -f docker/Dockerfile.arm64 --target export \ + --output type=local,dest=/tmp/vllm-arm64 . +``` + +This keeps source compilation, build packages and build trees off the Pi. The +exported glibc 2.39 binaries are deployed into a disposable user-owned Pi +directory and execute against the Pi's newer glibc. QEMU proves AArch64 build, +link and instruction execution, but its time and virtual PMU are never accepted +as Cortex-A76 performance evidence. All binding cycles, stalls, cache events, +wall time, frequency, temperature and throttling metadata come from execution +of the exported artifact on the physical Pi. + ## Upstream and dependency chain vLLM is the semantic oracle, not the Pi implementation source: @@ -83,7 +105,7 @@ The current production path is already shared and model-independent: | Paged attention | `src/vt/cpu/cpu_paged_attn.cpp` | correctness and 1/2/4-core scaling owed | | Qwen3.5 forward | `src/vllm/model_executor/models/qwen3_5*.cpp` | reuse unchanged; no Pi-private forward | | Provider selection | `include/vt/op_provider.h`, `src/vt/op_provider.cpp` | register A76 variants here with same-binary fallback | -| CPU microbench | `examples/cpu_kernel_bench/main.cpp` | R1 vt-op/PMU substrate CPU-gated; Pi execution pending | +| CPU microbench | `examples/cpu_kernel_bench/main.cpp` | R1 vt-op/PMU substrate CPU-gated; QEMU build and physical-Pi execution green | | Legacy quant evidence | `examples/quant_gemm_bench/main.cpp` | retained unchanged for prior results | Every operation observed in the Qwen trace is entered into the experiment @@ -175,8 +197,8 @@ matching llama.cpp quant/repack cases for any borrowed layout or kernel: 1. x86-64 captures fixed prompt IDs, seeds, sampling parameters, 16 greedy tokens, selected logits and operation fixtures from the pinned vLLM oracle and the current CPU implementation. -2. Pi portable build loads the exact-hash GGUF and matches 16/16 tokens before - any optimized provider is enabled. +2. The QEMU-built portable artifact loads the exact-hash GGUF on the Pi and + matches 16/16 tokens before any optimized provider is enabled. 3. Every provider matches its portable operation contract and the Pi full model repeats the cross-architecture golden. 4. Microbench A/B uses randomized interleaved trials. A retained metric-level @@ -197,8 +219,8 @@ matching llama.cpp quant/repack cases for any borrowed layout or kernel: |---|---|---|---| | R0 | Spike, Pi inventory and fixed model recipe | observed hardware facts | this spec + current record surfaces | | R1 | **CPU-GATED** general CPU kernel/PMU harness | existing quant bench | warning-clean build; JSON/CLI/timer/counter contract; 1/4-thread runs | -| R2 | Portable Pi bring-up and x86 goldens | R0 | exact hash, load, 16/16 tokens, operation fixtures | -| R3 | Qwen trace + recursive scope profiling | R1-R2 | complete reached-loop inventory and binding baseline | +| R2 | **GREEN** QEMU-built portable Pi bring-up and x86 goldens | R0 | exact hash, load, 16/16 tokens, operation fixtures | +| R3 | **GREEN** Qwen trace + recursive scope profiling | R1-R2 | reached-loop inventory and binding baseline below | | R4 | A76 C++/NEON/SDOT providers | R3 ranked evidence | op correctness + causal metric win + no enclosing regression | | R5 | A76 assembly candidates | R4 plateau + proven compiler gap | ABI/disassembly/correctness + recursive A/B | | R6 | Whole-system/thread/serving exhaustion | accepted R4/R5 stack | all lever dispositions, <1% residual model, llama.cpp floor | @@ -207,6 +229,31 @@ R1-R3 are the first implementation checkpoint. R4/R5 split into separate kernel-row PRs if the changed code exceeds the one-row helper size cap; this PR does not silently absorb unrelated kernel families. +## R2-R3 binding result + +The Ubuntu 24.04 buildx/QEMU build completed with GCC 13.3 and its quantized +matmul smoke passed. The two exported AArch64 binaries were hash-gated before +deployment. On the physical Pi, the pinned model matched the x86 current-engine +golden 16/16 tokens (golden file SHA-256 `684f55a32355c0ccb6ce9c987273981f077b9591a46db07aea68561eb6432966`). +The four portable fixture arms also retained their exact checksums: decode +M=1 `0xd6aec014c0050fda` and prefill M=128 `0xa89baff1f3a4e360`, at one and +four threads. + +The idle, unthrottled 2.4 GHz Pi baseline measured Q8_0 M=1/N=3072/K=2048 at +1,554,115 ns median (8.10 GFLOP/s) on one core and 742,585 ns (16.94 GFLOP/s) +on four. The M=128 arm measured 197,061,735 ns on one core and 49,890,756 ns +on four (3.95x scaling). The 16-token model arm measured TTFT 1,961.99 ms, +TPOT/ITL 366.91 ms and output throughput 2.14 tok/s. These are portable +baselines, not optimized results or llama.cpp parity claims. + +A zero-loss, low-overhead `cycles:u` profile of a 64-token model run ranks the +reached loops: BF16 `Bt16Neon` 57.76%, portable `VecDotQ8_0Q8_0` 20.10%, +thread-ready 6.45% and `F16ToF32` 4.87%. The Q8 dot is selected for R4/R5: +the Pi has DotProd but the only existing Arm quant fast path requires i8mm, so +the real model currently executes the scalar portable dot. The next checkpoint +must compare portable, compiler-generated exact-order SDOT and AAPCS64 assembly +in one binary before any default dispatch changes. + ## Risks and decisions - `BACKEND-CPU` already passes a 20-core Arm i8mm Qwen3.5-2B single-stream @@ -220,6 +267,7 @@ does not silently absorb unrelated kernel families. - PMU counter availability is verified, but permissions and event scheduling can still reject an event. The harness reports unsupported events and never fabricates zero counts. -- Temporary governor/tool installation and model transfer were approved by the - user. Every system setting is recorded and restored; no public service is - started and vLLM is not installed on the Pi. +- Artifact/model deployment into a disposable user-owned directory is approved. + Source is never compiled on the Pi. Every temporary system setting is + recorded and restored; no public service is started and vLLM is not installed + on the Pi. diff --git a/.agents/state.md b/.agents/state.md index 204c31d4c..6af35f989 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39333,3 +39333,34 @@ invalid-input and structured-counter cases. Direct x86 executions passed at groups returned real counts. Those x86 timings are tool validation only and are not binding performance evidence. Pi PMU execution, Qwen correctness and llama.cpp comparison remain `PENDING`; next is R1-on-Pi, then R2 bring-up. + +## 2026-08-06T17:45 - BACKEND-CPU R2-R3 portable Pi baseline green + + +The CPU-only ARM64 build now runs entirely through local Docker buildx/QEMU; +the Pi remains execution-only. Ubuntu 24.04/GCC 13.3 built and linked +`vllm-bench` plus `vllm-cpu-kernel-bench`, and QEMU executed the quantized +matmul smoke. Exported binaries and the pinned 2.83 GB Q8_K_XL model were +SHA-256-gated before deployment into `~/vllm-cpp-assembly`. + +R2 correctness is GREEN. The Pi generated the exact x86 current-engine golden +over 16/16 tokens; the token file SHA-256 is +`684f55a32355c0ccb6ce9c987273981f077b9591a46db07aea68561eb6432966`. +Portable Q8 operation checksums were exact at one and four threads: decode +M=1/N=3072/K=2048 `0xd6aec014c0050fda`, prefill M=128 +`0xa89baff1f3a4e360`. + +R3's idle, unthrottled 2.4 GHz baseline measured the M=1 fixture at 1,554,115 +ns median / 8.10 GFLOP/s on one core and 742,585 ns / 16.94 GFLOP/s on four; +the M=128 fixture measured 197,061,735 ns and 49,890,756 ns (3.95x). The +16-token model arm measured 2.14 output tok/s, TTFT 1,961.99 ms and TPOT/ITL +366.91 ms. A separate 64-token `perf record -e cycles:u -c 1000000` captured +241K samples with zero loss: `Bt16Neon` 57.76%, portable +`VecDotQ8_0Q8_0` 20.10%, thread-ready 6.45%, `F16ToF32` 4.87%. + +The real model therefore reaches a material scalar Q8 dot hot spot. Existing +Arm acceleration requires i8mm, which the A76 lacks; DotProd is available. +R4/R5 will compare an exact-order C++ SDOT intrinsic against a scheduled +AAPCS64 implementation in one QEMU-built binary, using the portable checksums, +PMU fixture and full-model run as recursive gates. No speedup or llama.cpp +parity is claimed at this checkpoint. diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000..6f5f51fa5 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,5 @@ +build*/ +**/build*/ +.git/ +.hf-cache/ +*.gguf diff --git a/docker/Dockerfile.arm64 b/docker/Dockerfile.arm64 new file mode 100644 index 000000000..b513808a0 --- /dev/null +++ b/docker/Dockerfile.arm64 @@ -0,0 +1,10 @@ +ARG UBUNTU=24.04 +FROM ubuntu:${UBUNTU} AS build +ARG JOBS=4 +RUN apt-get update && apt-get install -y --no-install-recommends build-essential cmake ca-certificates && rm -rf /var/lib/apt/lists/* +WORKDIR /src +COPY . . +RUN --mount=type=cache,id=vllm-cpp-arm64-build,target=/build cmake -S . -B /build -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_SERVER=OFF && cmake --build /build -j${JOBS} --target vllm-bench vllm-cpu-kernel-bench && /build/examples/vllm-cpu-kernel-bench --m 1 --n 32 --k 32 --threads 1 --warmup 0 --iterations 1 --counters off --no-pin --format json && mkdir -p /out && cp /build/examples/vllm-bench /build/examples/vllm-cpu-kernel-bench /out/ +FROM scratch AS export +COPY --from=build /out/vllm-bench / +COPY --from=build /out/vllm-cpu-kernel-bench / diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 9ec022f92..11cc717e2 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -154,15 +154,20 @@ host mirror is freed once the device Marlin resident is built. ## llama.cpp, CPU -Raspberry Pi 5 Cortex-A76 is a separate `PENDING` arm. It has four cores, +Raspberry Pi 5 Cortex-A76 is a separate `GATING` arm. It has four cores, DotProd and no i8mm, so the binding 20-core Arm result below does not transfer. -Reproduction starts with the exact Q8_K_XL SHA-256 recorded in the -[RPi5 spike](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md), portable -16-token correctness, then three interleaved vllm.cpp/llama.cpp repetitions at -1/2/4 threads. The R1 vt-op harness is CPU-gated and emits PMU-accounted JSON; -the first Pi probe is -`vllm-cpu-kernel-bench --dtype q8_0 --m 1 --n 3072 --k 2048 --threads 1 --cache stream --counters a76 --format json`. -No Pi throughput or memory number is accepted yet. +The R2-R3 portable baseline uses the exact Q8_K_XL SHA-256 in the +[RPi5 spike](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md) and a local +buildx/QEMU ARM64 build; QEMU timing is non-binding. The exported artifact ran +unthrottled at 2.4 GHz on the Pi and matched the x86 golden 16/16 tokens. Q8_0 +M=1/N=3072/K=2048 measured 1,554,115 ns at one thread and 742,585 ns at four; +M=128 measured 197,061,735 ns and 49,890,756 ns respectively. Exact fixture +checksums held in every arm. The 16-token model arm measured TTFT 1,961.99 ms, +TPOT/ITL 366.91 ms and output throughput 2.14 tok/s. A zero-loss 64-token +`cycles:u` trace attributes 57.76% to BF16 GEMM and 20.10% to the portable Q8 +dot. These numbers bind the portable denominator only. SDOT/assembly A/B, +three interleaved full-model repetitions, peak memory and same-file llama.cpp +comparison remain `PENDING`. Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7a9a743dd..f751d4bd7 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; A76 PMU harness, SDOT gating) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; QEMU-built A76 correctness/PMU baseline, SDOT gating) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 5a8936732..f28c2738e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi (DotProd, no i8mm): R1 PMU/JSON vt-op harness CPU-gated; Pi execution, model bring-up and perf `PENDING`. Concurrency open | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi R2-R3: QEMU-built artifact is 16/16 x86-token exact and fixture-exact; unthrottled portable baseline captured, with Q8 dot 20.10% of user cycles. SDOT/assembly A/B and Pi llama.cpp floor remain open. Concurrency open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 797b542cc..879d504b7 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -51,8 +51,6 @@ target_link_libraries(quant-gemm-bench PRIVATE vllm::vllm) target_include_directories(quant-gemm-bench PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_set_warnings(quant-gemm-bench) -# General PMU-backed CPU operation harness (BACKEND-CPU R1). quant-gemm-bench -# stays available so the earlier binding evidence remains reproducible. add_executable(vllm-cpu-kernel-bench cpu_kernel_bench/main.cpp) target_link_libraries(vllm-cpu-kernel-bench PRIVATE vllm::vllm) target_include_directories(vllm-cpu-kernel-bench PRIVATE ${CMAKE_SOURCE_DIR}/src) From b1be2bb65cf8e96a60b7eb2f810f3ea2cbd7b551 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 17:51:46 +0100 Subject: [PATCH 5/7] Spike Cortex-A76 Q8 dot assembly row Inventory the measured Q8_0 DotProd opportunity, bind compiler-SDOT versus scheduled AAPCS64 gates, and claim the independent kernel row before implementation. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/coordination.md | 14 +++ .agents/kernel-matrix.md | 8 +- .agents/roadmap_v1.md | 2 +- .agents/specs/cpu-a76-q8-dot.md | 161 ++++++++++++++++++++++++++++++++ .agents/state.md | 16 ++++ docs/BENCHMARKS.md | 4 + docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- scripts/check-agent-record.py | 7 +- 9 files changed, 209 insertions(+), 7 deletions(-) create mode 100644 .agents/specs/cpu-a76-q8-dot.md diff --git a/.agents/coordination.md b/.agents/coordination.md index 64982d198..788c454cb 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,6 +118,19 @@ without the selected contention proof for their entire run are discarded. ## Active claims +**Cortex-A76 Q8 SDOT/assembly (`KERNEL-CPU-A76-Q8-DOT`, 2026-08-06, +`CLAIM-KERNEL-CPU-A76-Q8-DOT`).** Codex (GPT-5), stacked local branch +`row/KERNEL-CPU-A76-Q8-DOT` in isolated worktree +`/tmp/vllm-cpp-row-BACKEND-CPU`, base `BACKEND-CPU` R2-R3 commit `16c848326`. +Owns only the new A76 Q8 dot C++/assembly TUs, their `include/vt/quant.h` and +CMake wiring, the Q8-specific portions of `cpu_quant_dot.cpp`, +`test_ops_quant_dot.cpp` and `examples/cpu_kernel_bench/main.cpp`, plus this +claim, `KERNEL-CPU-A76-Q8-DOT`, its spike and required checkpoint surfaces. +Builds ARM64 only through local buildx/QEMU; deploys exported artifacts into +the user-authorized disposable `rich@rpi5fan.lan:~/vllm-cpp-assembly` for +uncontended correctness/PMU A/B. No Pi compilation, system/service mutation, +other quant format, BF16 kernel, model-forward or remote publication ownership. + **Interactive CLI chat/complete spike (`SERVE-CLI-CHAT`, 2026-08-01, `CLAIM-SERVE-CLI-CHAT-SPIKE`).** Codex (GPT-5), isolated worktree `/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`, @@ -1377,6 +1390,7 @@ table, tests, CMake. Details in the state-log entry of the same date. | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| +| `CLAIM-KERNEL-CPU-A76-Q8-DOT` | `KERNEL-CPU-A76-Q8-DOT` | Codex (GPT-5) | isolated worktree `/tmp/vllm-cpp-row-BACKEND-CPU`; local QEMU/buildx builder `pf-arm`; execution-only `rich@rpi5fan.lan:~/vllm-cpp-assembly` | `row/KERNEL-CPU-A76-Q8-DOT`, stacked on `16c848326` | New A76 Q8 C++/assembly TUs, narrow Q8 dispatch/header/CMake/test/harness edits, spike and checkpoint records. No Pi compilation, other quant/BF16/model-forward work or remote publication | `ACTIVE` | 2026-08-06, W0 spike; implementation and physical-Pi A/B next | | `CLAIM-KIMI-LINEAR-W0` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — dedicated W0 spike) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a771cc029e6843be0`; CPU-only, records-only — NO build, NO GPU, NO download (two GPU jobs queued ahead; the W0 GPU golden capture is a SEPARATE later step) | branch `worktree-agent-a771cc029e6843be0`, base `origin/main` HEAD `10dd23ee` | The FULL dedicated W0 spike for `KimiLinearForCausalLM` (Kimi-Linear-48B-A3B) per the spike-first protocol, so W1 implementation can start immediately. Owns ONLY: NEW `.agents/specs/kimi-linear.md`, the `MODEL-TEXT-kimi-linear-*` matrix row (Spike link → the dedicated spec + note; STAYS `SPIKE` because it is actively claimed — protocol forbids a claimed row from being `READY`; checklist mark stays `📋`), this claim row + the narrative block below, the roadmap breadth note, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, the `.agents/NOW.md` live-claim row + stamp, and one `.agents/state.md` entry. **NON-COLLISION:** records-only — touches NO model/kernel/registry/loader source, NO CMake, NO test. Co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` (MLA half) + `CLAIM-KDA-KERNEL` (KDA host refs), exactly as the K3 row is co-owned by `CLAIM-KIMI-K3-SCOPE`/`CLAIM-KIMI-K3-W2-W5`. | `DONE` | 2026-08-05 — **W0 DEDICATED SPIKE LANDED (records-only, NOT pushed).** Authoritative `config.json` fetched (27 layers = 20 KDA + 7 NoPE-MLA; 256e/top-8/1-shared sigmoid `noaux_tc` `routed_scaling=2.446` `first_k_dense_replace=1`; `num_nextn_predict_layers=0` ⇒ no MTP); reuse-vs-new map with our `file:line`; HW-fit (FITS one GB10, 91.5 GiB / 0.77× pool — real e2e SACRED gate reachable); W0-W7 breakdown + GPU golden-capture recipe. NET-NEW = KDA device kernel (host refs landed, `CLAIM-KDA-KERNEL`), NoPE-MLA branch, hybrid schedule/het-KV, loader name-map. Record checkers green (`check-model-checklist`, `check-agent-record`, `check-doc-checkpoint`, `check-now-current`). | | `CLAIM-KIMI-LINEAR-W1` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W1 registry/config/loader scaffolding; forward REFUSES-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a11ae231a71e8694b`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release); NO GPU, NO download — additive TU + one REGISTER line, forward is a `VT_CHECK(false)` stub | branch `worktree-agent-a11ae231a71e8694b`, base `origin/main` HEAD `053116df` | Kimi-Linear-48B-A3B W1 — registry + config + loader + KV-spec scaffolding so the W3-W6 forward can start. Owns ONLY: NEW `include/vllm/model_executor/models/kimi_linear.h`, NEW `src/vllm/model_executor/models/{kimi_linear_registry,kimi_linear_weights,kimi_linear}.cpp`, NEW `tests/vllm/models/test_kimi_linear_scaffold.cpp`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `KimiLinearForCausalLM` sorted-set + error-message + model-property inserts in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the forward refuses, rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` refuse stub so NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W1 CPU SCAFFOLDING LANDED (foreground, NOT pushed).** Additive registry (`kimi_linear`/`KimiLinearForCausalLM`, `is_hybrid`, `supports_multimodal=false`) + `ParseKimiLinearParams` (20 KDA + 7 NoPE-MLA schedule, MLA 512/128/64/128 q_lora-null, 256e/top-8/1-shared sigmoid `noaux_tc` 2.446, `first_k_dense_replace=1`; asserts `mla_use_nope`/`q_lora==null`) + `EnumerateKimiLinearTensors` name-map VERIFIED vs the real HF safetensors index (MoE = `block_sparse_moe.*`, the correction of the DERIVED K3 `mlp.*`) + loader THROWS BY NAME on missing/mis-shaped tensors + het KV spec (MLA latent-576 + KDA mamba `12288×3`/`32×128×128`) + REFUSE-by-name forward. `test_kimi_linear_scaffold` **9/9·83** + `test_model_registry` **24/24** + `test_kimi_k3_scaffold` 6/6 unchanged; clean CPU Release build; `check-fusion-consistency`/`check-runner-routing-consistency` green (refuse stub skipped); record checkers rc=0. RESIDUAL (W3-W6): KDA device kernel + NoPE-MLA route + sigmoid-noaux MoE + het-KV born-on-runner forward + the e2e SACRED gate (spec §8 recipe). | | `CLAIM-KIMI-LINEAR-W2` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W2-W6 CPU REFERENCE forward; the DEVICE runner forward stays refuse-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-aec13af094cd4692f`; CPU-only `build` (`-DVLLM_CPP_CUDA=OFF` RelWithDebInfo); NO GPU, NO download — additive forward TU + a host-materialization pass in the existing loader + additive test | branch `laguna-s21-w7-speed-profile` (worktree), base `origin/main` HEAD `0988db48` | Kimi-Linear-48B-A3B W2-W6 — the CPU REFERENCE forward, per-op gated, replacing the refuse-by-name host `KimiLinearModel::Forward`. Owns ONLY: NEW `src/vllm/model_executor/models/kimi_linear_forward.cpp`, NEW `tests/vllm/models/test_kimi_linear_forward.cpp`, its two CMake lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), ADDITIVE host-weight structs + per-op decls in `include/vllm/model_executor/models/kimi_linear.h`, the host-materialization pass in `src/vllm/model_executor/models/kimi_linear_weights.cpp`, the device-refuse doc/rename in `src/vllm/model_executor/models/kimi_linear.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the DEVICE forward refuses; rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + additive header structs; the host `Forward` becomes real but `ForwardDevice` (the DEFAULT gather_logits runner path) STAYS `VT_CHECK(false)` so the born-on-runner CI guard skips it (NO allowlist) and NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0/W1`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W2-W6 CPU REFERENCE FORWARD LANDED (foreground, NOT pushed).** Real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives: KDA layer (`vllm::kimi_kda` refs + the gated-delta recurrence `fused_recurrent.py:122-149`), NoPE-MLA (materialized-MHA reference, scaling `qk_head**-0.5`, no RoPE), sigmoid `noaux_tc` MoE (bias-select/unbiased-weight, renormalize, `routed_scaling=2.446`) + shared expert, dense layer-0 SwiGLU; loader now MATERIALIZES the host float weights (bf16/f32→f32). `test_kimi_linear_forward` **6/6·246** (KDA==kimi_kda composition; NoPE-MLA==materialized-MHA ref; router+block==hand-computed; loader materializes; whole 2-layer forward finite/coherent; greedy-decodes 5 tokens, context advances) + `test_kimi_linear_scaffold` 9/9·83 + `test_kimi_kda` 14/14 UNCHANGED; clean CPU build; `check-fusion-consistency`/`check-runner-routing-consistency` green (DEVICE forward refuse-skipped, no allowlist). RESIDUAL: the DEVICE born-on-runner forward + the W0/W7 e2e SACRED token golden on GB10 (spec §8 recipe) + speed. | diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index 1f7759ca8..e768a70af 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -112,6 +112,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner | |---|---|---|---|---|---|---|---| +| `KERNEL-CPU-A76-Q8-DOT` | Cortex-A76 Q8_0 x Q8_0 SDOT, exact-order C++ and scheduled AAPCS64 variants for the Pi decode/prefill keep-quant GEMM | llama.cpp `ggml/src/ggml-cpu/quants.c:400` portable dot and `arch/arm/quants.c:1076-1160` Arm DotProd @ project pin `237ad9b96`; Qwen semantics remain vLLM `555967922` | Baseline [portable dot](../src/vt/cpu/cpu_quant_dot.cpp#L88) and [quant GEMM caller](../src/vt/cpu/cpu_quant_gemm.cpp#L72); optimized provider not implemented at spike | Existing [Q8 upstream-test port and quant gates](../tests/vt/test_ops_quant_dot.cpp#L559); R3 zero-loss Pi model trace reaches portable Q8 dot at 20.10% of user cycles; assembly evidence pending | [A76 Q8 dot spike](specs/cpu-a76-q8-dot.md) | `SPIKE` | `CLAIM-KERNEL-CPU-A76-Q8-DOT` | | `KERNEL-ACCEL-PROVIDER-SELECT` | **WHICH implementation of an op runs, when more than one exists on a device** — the selection layer above every kernel family in this matrix. Distinct from `KERNEL-CUDA-DISPATCH-AOT`, which is about which ARCH a CUDA kernel is compiled/selected for; this is about which PROVIDER (ours, a vendor library, MLX, llama.cpp) serves the op at all | no single upstream file: this is the shape vLLM's runtime chain uses everywhere — flashinfer tactic registries, cuBLASLt/CUTLASS per-call heuristics, and torch's backend selection — rather than compile-time pinning | `vt::OpProvider` [op_provider.h](../include/vt/op_provider.h) + [op_provider.cpp](../src/vt/op_provider.cpp); the flat `[OpId][DeviceType]` `void*` table it replaces is gone from [ops.cpp](../src/vt/ops.cpp) with the ~70 op wrappers untouched. Providers registered today: `vt-native` (every backend kernel in the tree, priority 0, unconditional — behaviour preserved exactly), `mlx` (priority 100, Metal `kMatmul`/`kMatmulBT`, build-gated `VLLM_CPP_MLX`, [metal_mlx_provider.mm](../src/vt/metal/metal_mlx_provider.mm)), and — NEW 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`, work row `S5`) — **`vt-cpu-ref` (priority −1000, the portable reference tier)**: the CPU kernel installed LAZILY as a negative-priority fallback on a UNIFIED-MEMORY device's first `GetOp` miss, mirroring `custom_op.py:138 forward_native`, so a partial backend runs an op it lacks natively instead of throwing. Native always wins (priority); gated on `Backend::UnifiedMemory()` (a discrete GPU never gets it — a CPU kernel on true device memory is corruption); observable via `GetReferenceTierHits()` + a one-time loud stderr line | [test_op_provider.cpp](../tests/vt/test_op_provider.cpp) 11 cases / 47 assertions — deterministic selection under REVERSED registration order, name tie-break, duplicate rejection, capability predicate, caps re-resolution, decline-and-fall-back, stats, runtime disable; [test_metal_backend.cpp](../tests/vt/test_metal_backend.cpp) 9 cases / 108 assertions on the M4 with MLX ON, including MLX-vs-MSL-vs-CPU NMSE per op at real shapes and an end-to-end DECLINE; **[test_reference_tier.cpp](../tests/vt/test_reference_tier.cpp) (S5): discrete-device refusal + unified-device zero-native-kernel fallback correctness + native-wins + observability, hardware-free via a fake backend on `kXPU`.** Linux CPU 156/156; dgx regression set ALL UNCHANGED — anchor `tests/vt/test_op_provider.cpp:64` | [Metal/MLX reuse study §6](specs/metal-mlx-reuse-study.md); [accelerator-seam-audit §10](specs/accelerator-seam-audit.md); [drop-in kernel ABI](specs/dropin-kernel-abi.md) (the complementary ARGUMENT half) | `ACTIVE` — mechanism landed and gated with THREE provider kinds (`vt-native`, `mlx`, `vt-cpu-ref`); the CUDA/CPU/Vulkan vendor provider rows it was designed for are not yet populated (so the row is deliberately left open) | `CLAIM-BACKEND-ACCEL-PROVIDER-1` | | `KERNEL-CUDA-DISPATCH-AOT` | CUDA runtime, streams, graphs, per-arch dispatch, and generated/AOT artifact selection | platform selection `vllm/platforms/cuda.py:205-493`; gencode handling `CMakeLists.txt:201-220`; JIT packages `cmake/external_projects/triton_kernels.cmake:1-28` | [cuda_backend.cu:20](../src/vt/cuda/cuda_backend.cu#L20), [CMakeLists.txt:37](../CMakeLists.txt#L37), [TritonAOT.cmake:57](../cmake/TritonAOT.cmake#L57) | [CUDA backend tests](../tests/vt/test_cuda_backend.cpp#L31); the Triton GDN AOT tree is now vendored per-arch — `sm_121a` (RUNTIME-VERIFIED) plus `sm_80/86/89/90a/100a` (DERIVED+BUILD-VERIFIED, `CLAIM-TRITON-AOT-PER-ARCH`, 2026-07-28), each a real per-target cubin (`cuobjdump sm=80/86/89/90/100`); the CUDA fast-path arch fan-out (non-GDN) remains as evidenced in `backend-matrix.md` | [inventory](specs/kernel-family-inventory.md); [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `ANCHOR-BACKFILL` | - | | `KERNEL-FUSION-FRAMEWORK` | **Portable automatic op-fusion framework (roadmap_v1 ORDER-1 extensibility cornerstone)** — declare-once `constexpr FusedRecipe` catalog (backend-agnostic, above `vt::`) transcribing vLLM's finite fusion-pass set; realize-per-backend through the `vt::` op table (Tier-0 composite = CPU oracle inherited free; Tier-1 interpreter = one kernel/backend lights up every recipe). A new vLLM fusion PR = ONE declaration; a new GPU = one catalog realization; a new model = additive pattern bindings. NOT a perf lever (35B ceiling ~3.5%/step compute-bound); primary value = extensibility + mechanical upstream-sync + CPU/CUDA oracle-drift elimination | pass manager `vllm/compilation/passes/pass_manager.py:86,138-200`; the finite fusion-pass set `passes/fusion/{rms_quant_fusion.py:618,act_quant_fusion.py:283,qk_norm_rope_fusion.py:188,attn_quant_fusion.py:362}`; declare-once analog `vllm/model_executor/custom_op.py:103` (`forward_native`/`forward_cuda`/`enabled()` :138-207); combo_kernels OUT-OF-SCOPE `vllm/config/compilation.py:967` | Phase-0 skeleton LANDED: [fused_recipe.h:72](../include/vt/fused_recipe.h#L72), [recipes.h:35](../include/vt/recipes.h#L35), [kFusedChain dispatch](../src/vt/ops.cpp#L639), Tier-0/1 CUDA [cuda_ops.cu:1062](../src/vt/cuda/cuda_ops.cu#L1062) + CPU [cpu_ops.cpp:1427](../src/vt/cpu/cpu_ops.cpp#L1427); catalog-to-be = the hand-fused ops (`kRmsNormQuantFp8`/`kSiluMulFp4Quant`/`kAttnQkNormRopeGate`/`kSigmoidGateFp4Quant`/`kRmsNormGatedQuantFp8`). **2026-07-19 (`CLAIM-FUSION-FRAMEWORK-W0`): W0 ADOPTED at ONE production site** — `RunLayerPaged` post_attention_layernorm (35B MoE paged, `test_qwen36_paged_engine` hot path) now routes the plain add+residual+gemma-RMSNorm through [`vt::FusedChain(kFusedAddRmsNorm)`](../src/vllm/model_executor/models/qwen3_5.cpp#L4784) behind [`FusedChainAdoptEnabled()` (`VT_FUSED_CHAIN_ADOPT`, default ON / `=0` restores the exact prior `vt::RmsNorm(residual)` hand-call)](../src/vllm/model_executor/models/qwen3_5.cpp#L1394). BEHAVIOR-PRESERVING by construction (Tier-0 composite dispatches to the same `vt::RmsNorm(residual)` primitive; perf-neutral, NOT a speedup) — proves the declare-once/realize-per-backend seam end-to-end in production. The other 3 identical plain sites (`RunLayer`/`RunDenseLayer`/`RunDenseLayerPaged` post-attn + the input-layernorm plain path) are left on the hand-call; W1's generalized POD (multi-input operands + quant/silu/rope opcodes) is required only for the quant-fused sites (`kSiluMulFp4Quant`/`kRmsNormQuantFp8`/…), NOT for these plain add+rmsnorm sites — the current 3-opcode/4-role POD expresses them byte-identically. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W1`): W1 GENERALIZED the POD (infrastructure only, perf-neutral, NO call site changed).** `FOp` grew from `{kAdd,kMul,kRmsNorm}` to add `kSilu`/`kSigmoid` (Tier-1 elementwise) + fused-primitive opcodes mapping 1:1 to standalone ops so the composite is byte-exact by construction — `kSiluMul`→`vt::MoeSiluMul`, `kSigmoidGate`→`vt::SigmoidGateBf16`, `kRmsNormGated`→`vt::RmsNormGated`, `kRope`→`vt::RopeFromCache`, `kQuantFp8`→`vt::QuantFp8Static`, `kQuantFp4`→`vt::ScaledFp4Quant`, `kAttnQkNormRopeGate`→`vt::AttnQkNormRopeGate`. The fixed `{kIn,kResidual,kWeight,kOut}` 4-role model became an INDEXED `FOperandSlot operands[8]` table (`FStep` names input indices + `out`/`out2`); tensors bind positionally via `FusedBinding`, scalars via `FusedParams`; intermediates are caller-bound scratch (byte-exact to the unfused sequence). Tier-0 composite is now ONE device-agnostic walker in `ops.cpp` (per-opcode dispatch to the standalone op, self-dispatching per device — eliminates CPU/CUDA oracle drift); the per-backend `kFusedChain` op is now ONLY the Tier-1 interpreter over the canonical `[x,weight,residual,out]` shape. All five W2 target chains DECLARED as `constexpr` recipes ([recipes.h](../include/vt/recipes.h), citing each vLLM pass). TIER (honest): composite realizes all five byte-exact; Tier-1 covers only the elementwise/rmsnorm subset (W0 `kFusedAddRmsNorm`) — the four quant/activation chains + attn preamble are COMPOSITE-ONLY. Backend-negotiation (§3b/§6): fp8 terminal `vt::QuantFp8Static` CUDA-only (fp8 chains' composite CUDA-only; CPU covers up to the bf16 norm); `kAttnQkNormRopeGate` a composite-only MACRO (per-head 3-D operands → dispatches the whole preamble to the single `vt::AttnQkNormRopeGate` op). ALL FIVE byte-exact expressible, none forced. Anchors: generalized POD [fused_recipe.h](../include/vt/fused_recipe.h), binding/params/general entry [ops.h](../include/vt/ops.h), device-agnostic composite [`FusedChainComposite`](../src/vt/ops.cpp), Tier-1 interpreters [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)+[cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W2`): W2 MIGRATED the bespoke hand-fusions to `vt::FusedChain(recipe)` — the framework now OWNS the fusion dispatch, perf-neutral by construction.** New realization-binding infra (the "fast realization" tier the spike §3c reserved): `FusedRecipe.fast_op` ([fused_recipe.h](../include/vt/fused_recipe.h)) carries the raw `OpId` of the EXISTING single-launch bespoke fused kernel that realizes the whole recipe (backend-AGNOSTIC — names WHICH abstract op, realized per-backend via the op table); a non-throwing `OpRegistered` probe + a per-recipe `DispatchFusedFast` adapter switch + convenience overloads in [ops.cpp](../src/vt/ops.cpp)/[ops.h](../include/vt/ops.h). `FusedChain(recipe,…)` dispatches recipe→fast_op WHEN the backend registers that OpId (the SAME kernel the model called directly pre-migration → byte-identical AND perf-neutral; no extra kernel, no per-forward getenv/alloc), else falls through to the byte-exact Tier-0 composite (graceful degradation). The composite is the exposed oracle ([`FusedChainComposite`](../src/vt/ops.cpp)) the fast realization is validated against. Bound recipes: `kRmsNormQuantFp8`→`OpId::kRmsNormQuantFp8`, `kRmsNormGatedQuantFp8`→`OpId::kRmsNormGatedQuantFp8`, `kSiluMulFp4Quant`→`OpId::kSiluMulFp4Quant`, `kSigmoidGateFp4Quant`→`OpId::kSigmoidGateFp4Quant` ([recipes.h](../include/vt/recipes.h)); `kAttnQkNormRopeGate` has NO fast_op (its composite MACRO already dispatches to the single `vt::AttnQkNormRopeGate` launch — perf-neutral trivially). SIX call sites migrated in [qwen3_5.cpp](../src/vllm/model_executor/models/qwen3_5.cpp) behind `VT_FUSED_CHAIN_ADOPT` (`=0` restores the exact prior hand-calls, same binary): `kSiluMulFp4Quant` (MoE down-proj), `kSigmoidGateFp4Quant` (full-attn o-proj), `kRmsNormGatedQuantFp8` ×2 (GDN out-proj), `kRmsNormQuantFp8` (input-layernorm), `kAttnQkNormRopeGate` ×2 (attn preamble). qwen3_5.cpp net +61 lines (−21/+82): each migrated fused-op call became one `FusedChain` call (call-for-call neutral), but the mandated same-binary rollback `else` branch + explanatory comment at each of the 6 sites grows the raw count — the true shrinkage lands when the rollback branches and bespoke `OpId`s retire; the dispatch surface is already framework-OWNED. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W3`): W3 MECHANICAL-UPSTREAM-SYNC PROOF** — ported a NEW, previously-unported vLLM fusion-pass variant, `SiluMulFp8StaticQuantPattern` (`vllm/compilation/passes/fusion/act_quant_fusion.py:81` → `_C.silu_and_mul_quant`; registered unconditionally at `act_quant_fusion.py:296`), as ONE `constexpr FusedRecipe kSiluMulQuantFp8` ([recipes.h](../include/vt/recipes.h)) + its byte-exact test. The static-per-tensor-FP8 sibling of `kSiluMulFp4Quant`; its Tier-0 composite is realized ENTIRELY from EXISTING standalone `vt::` ops (`kSiluMul`→`vt::MoeSiluMul`, then `kQuantFp8`→`vt::QuantFp8Static`) — NO new primitive, NO kernel, NO composite-walker case, NO dispatch edit, NO model-site edit. `fast_op = kNoFastOp` (no bespoke silu·mul→static-fp8 fused OpId exists in-tree; a fast kernel is a separate later perf step). **Additivity headline: adding a whole new fusion pattern touched exactly 2 shared files** — `include/vt/recipes.h` (+1 declaration) + `tests/vt/test_ops_fused_chain.cpp` (+1 byte-exact test); `git diff --stat main` = those 2 files only. This IS the framework's primary-value proof (spike §4/§10): a new upstream fusion PR = one declaration. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W4`): W4 — the EXECUTABLE BACKEND-ADDITIVITY PROOF, closing the W-series.** Makes the claim executable: "a new backend registers `kFusedChain` (Tier-0 composite) ONCE and inherits the ENTIRE catalog correct, ZERO per-recipe work." Proof approach (spec §10 W4, PREFERRED option b): treat the EXISTING CPU backend AS the 'second backend' relative to CUDA — NO mock `DeviceType` (which would edit the core enum + every switch, ironically non-additive). New test [`test_fused_chain_additivity.cpp`](../tests/vt/test_fused_chain_additivity.cpp) enumerates the WHOLE catalog (`kCatalog`, all 7 recipes) and in ONE generic loop asserts each runs correct on the CPU backend via the Tier-0 composite ([`vt::FusedChainComposite`](../src/vt/ops.cpp)), BYTE-EXACT vs the standalone-op-sequence golden, over the CPU-expressible scope: 4 CPU-FULL end-to-end (`kFusedAddRmsNorm`/`kSiluMulFp4Quant`/`kSigmoidGateFp4Quant`/`kAttnQkNormRopeGate`), 3 CPU-PREFIX (`kRmsNormQuantFp8`/`kRmsNormGatedQuantFp8`/`kSiluMulQuantFp8` — static-fp8 terminal `vt::QuantFp8Static` CUDA-only per §3b/§6; the prefix is byte-exact and the FULL composite is asserted to THROW on CPU, documenting the backend-negotiated tail, not silently skipping). ADDITIVITY EVIDENCE (catalog GROWS ⇒ backend does NOT): catalog [recipes.h](../include/vt/recipes.h) 1→6→7 recipes (W0→W1→W3); the composite walker [`FusedChainCompositeImpl`](../src/vt/ops.cpp) is ONE function with a per-OPCODE switch (12 `FOp::` cases, NOT per-recipe); the CPU/CUDA `kFusedChain` registration is ONE line each; `src/vt/cpu/cpu_ops.cpp` does NOT `#include recipes.h` (the backend never sees the catalog); W3's whole new recipe `kSiluMulQuantFp8` appears in ZERO backend TUs + ZERO in the `ops.cpp` dispatch — it runs purely through the shared walker (that is WHY W3 touched 2 files and no `src/vt/`). **KERNEL-FUSION-FRAMEWORK ORDER-1 milestone DONE** (W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity). HONEST DEFERRED / future (named, non-blocking the ORDER-1 milestone): (a) the Tier-1 PERF interpreter covers only the elementwise/rmsnorm subset — the quant/activation chains realize composite-only (correct/byte-exact, but the single-pass fused perf kernel for the quant chains is future); (b) a REAL Metal/Vulkan realization of the catalog needs the M4 dev-box (HW-blocked); (c) per-recipe fast single-launch kernels for composite-only recipes are separate later perf steps. Perf is NOT the framework's value (§11: 35B ceiling ~3.5%/step, compute-bound). **2026-07-25 (`CLAIM-FUSION-CONSISTENCY-AUDIT`): CONSISTENCY AUDIT + ENFORCEMENT** — read-only static audit of catalog USAGE across all 12 model forwards ([fusion-consistency-audit.md](specs/fusion-consistency-audit.md)). Verdict MOSTLY consistent: qwen3_5 family fully migrated; qwen3/qwen3_moe/deepseek_v2 adopted `kFusedAddRmsNorm{,Std}`; **DRIFT — gemma/gemma2/gemma3/glm4/phi3 hand-call the residual `vt::RmsNorm(...,&res)` add+RMSNorm chain and never route through the catalog** though the recipe already exists (one-line adoption each = follow-on `FUSION-DENSE-MIGRATE`). olmo2/granite/opt deliberately-not-fused (post-norm/LayerNorm). GDN glue + MoE-combine remain bespoke (spike §10 W2 remainder). NEW enforcement CI check [`scripts/check-fusion-consistency.py`](../scripts/check-fusion-consistency.py) (+ mutation test, wired into the `agent-record` CI job) flags any model that hand-fuses add+RMSNorm without the catalog unless allowlisted — the 5 drift models on [`fusion-consistency-allowlist.txt`](../scripts/fusion-consistency-allowlist.txt) keep the gate GREEN while blocking new silent bypasses; removing a stem is the gate closing. | [test_ops_fused_chain.cpp](../tests/vt/test_ops_fused_chain.cpp) (byte-exact composite==interp==golden; W1 extends it: each new recipe's Tier-0 composite == its standalone-op-sequence golden byte-exact, fp4+attn CPU+CUDA, fp8 CUDA-only. **W2 adds the fast==composite==golden triple arm per recipe — CPU 228 assertions, CUDA-on-dgx 420, memcheck 0**). **W3 adds `RunSiluMulQuantFp8Cuda` (kSiluMulQuantFp8: FusedChain==composite==MoeSiluMul+QuantFp8Static golden, byte-exact, CUDA — the newly-ported pass's test), and the no-token-regression gate `test_qwen27_paged_engine` 235/235 + `test_qwen36_paged_engine` 315/315 unchanged (the new recipe is DECLARED, not wired into any model — the engine is untouched)**; **W4 adds `test_fused_chain_additivity.cpp`** (the whole-catalog additivity loop: every recipe byte-exact on the CPU 'second backend' via the Tier-0 composite, CPU-full end-to-end / fp8-prefix + negotiated-tail-throw; count guard `==7`) — **1 case / 17 assertions CPU green, sibling `test_ops_fused_chain` 228/228 unchanged, engine byte-identical so 27B 235/235 + 35B 315/315 structurally unchanged**; to port: `tests/compile/passes/test_fusion.py:249-287` oracle discipline (byte-exact, stricter than upstream `assert_close`) — anchor `tests/vt/test_ops_fused_chain.cpp:124` | [portable-fusion-framework.md](specs/portable-fusion-framework.md) | `ANCHOR-BACKFILL` (W-series ORDER-1 PROOF MILESTONE DONE: W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity, all merged+gated; row stays SPIKE-open because the framework retains named future work — the Tier-1 quant-chain perf interpreter (composite-only today) + a real Metal/Vulkan catalog realization (M4 HW-blocked) + per-recipe fast kernels, §10 Wn) | `CLAIM-FUSION-FRAMEWORK-W4` | @@ -160,9 +161,10 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. ## Count invariants -- This table has exactly 32 practical kernel-family rows. -- Baseline lifecycle counts are 8 `ANCHOR-BACKFILL`, 0 `READY`, 4 `PARTIAL`, - 6 `ACTIVE`, 0 `GATING`, 1 `DONE`, and 13 `INVENTORIED`. +- This table has exactly 46 stable kernel-family rows, including the distinct + `KERNEL-CPU-A76-Q8-DOT` compiler/assembly family added on 2026-08-06. +- Lifecycle counts are checked from the table itself by + `scripts/check-agent-record.py`; prose here is not a second status ledger. - The prefill/scratch GDN `ACTIVE` rows are valid only while `CLAIM-PR3` remains active; the FA2 row is valid only while `CLAIM-SERVE-GATE-1` claims it in `.agents/coordination.md`. `KERNEL-GDN-PACKED-DECODE` closed `DONE` diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 64ed24d45..0ec7c7ef0 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -68,7 +68,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 7 | `ROAD-V1-C7` | Sampling/API controls and logprobs payloads | [engine matrix](engine-matrix.md), [coverage view §6](feature-matrix.md#6-sampling--generation-controls), [C7 spec](specs/sampling-controls-c7.md) | **W1-W4 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7`, NOT pushed): the full sampling-control surface is WIRED end-to-end + gated exactly on the CPU reference backend.** `SAMPLE-CORE` + `SAMPLE-LOGIT-FILTERS` -> `ACTIVE`: SamplingParams undefers + validates logit_bias/allowed_token_ids/bad_words + all_stop_token_ids; OpenAI protocol parses the three fields + clamps logit_bias to [-100,100]; InputBatch tracks + `build_sampling_metadata` populates min_p/min_tokens/logit_bias/allowed_token_ids_mask/bad_words/max_num_logprobs with condense/swap maintenance; InputProcessor tokenizes bad_words + seeds the eos stop-set. RED-first (disabling a wiring line fails the gate); inertness proven (default/greedy build byte-identical SamplingMetadata, SACRED greedy unaffected); CPU `-Werror` 0-warn. The transforms are device-neutral pure functions gated exactly (not near-tie). **W5 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7-LOGPROBS`, NOT pushed): the logprobs PAYLOAD is wired end-to-end + gated on the CPU reference engine.** `SAMPLE-LOGPROBS` -> `DONE`: ported `vllm/logprobs.py` + `vllm/v1/engine/logprobs.py` (LogprobsProcessor); threaded ModelRunnerOutput.logprobs -> scheduler slice_request -> EngineCoreOutput.new_logprobs -> OutputProcessor -> CompletionOutput.logprobs -> OpenAI `CompletionLogProbs`/`ChatCompletionLogProbs` serialization (grounded 1:1 in vLLM 0.26 `555967922`). Gate: serialization vs a hand-computed vLLM oracle (completion N+1 vs chat N cutoff, bytes, text_offset, -9999 floor) RED-first, + e2e through the CPU Qwen3.6 engine (`logprobs=K` greedy sampled==argmax==rank1, chat `top_logprobs`, null-when-off inertness); CPU `-Werror` 0-warn. **C7-core (SAMPLE-CORE + SAMPLE-LOGIT-FILTERS + SAMPLE-LOGPROBS) is complete.** | `DONE` | **REMAINING (separate rows, NOT required for C7-core):** `SAMPLE-PROMPT-LOGPROBS` payload/serialization is done but its runner prompt-position-logits SOURCE is a runner/prefill addition (row `PARTIAL`); plus INVENTORIED `n>1` execution, `SAMPLE-PHILOX` exact RNG, `SAMPLE-LOGPROB-TOKEN-IDS`/logprobs_mode, beam search. **Custom logits processors LANDED 2026-07-28 (`SAMPLE-CUSTOM-PROCESSORS` -> `ACTIVE`, `CLAIM-C7-CUSTOM-LOGITS`): per-request C-ABI callback (`vllm_logits_processor`, ABI v8) at the sampler's non-argmax-invariant stage; default byte-identical; also closes SGLang `SGLANG-SAMPLING-CUSTOM`.** **Parallel sampling `n>1` LANDED 2026-07-28 (`SAMPLE-N` -> `ACTIVE`, `CLAIM-C7-N-SAMPLING`): the OpenAI `n` parameter now EXECUTES — an `n>1` request fans out into n prompt-sharing child sequences (`ParentRequest`/`FanOutParallelSampling`, mirroring vLLM `parallel_sampling.py`) aggregated into one `RequestOutput` / n indexed `choices`; `n==1` byte-identical; CPU-gated RED-first (1→n) with a `top_k=1` determinism gate. Residuals: `best_of`, async-streaming per-child collation, C-ABI `n`.** **Beam search LANDED 2026-07-28 (`SAMPLE-BEAM` -> `ACTIVE`, `CLAIM-C7-BEAM`): beam search as an OUTER engine loop (mirroring vLLM `entrypoints/generate/beam_search/{utils,offline}.py`) — per step one decode per beam (`logprobs=2*beam_width`), expand + score by `get_beam_search_score` (cum_logprob / `seq_len**length_penalty`), keep top-`beam_width`, retire EOS, return `beam_width` outputs (reuses the `SAMPLE-N` multi-output seam). Deterministic ⇒ token-EXACT: model-free gate vs a hand-computed beam tree (RED-first 1→2 beams) + e2e over the CPU engine (`bw=1`==greedy). Residuals: OpenAI `use_beam_search`/`best_of` wiring, C-ABI beam params, grammar-constrained beams.** **`best_of` + `use_beam_search` OpenAI-ENDPOINT surface LANDED 2026-07-28 (`SAMPLE-BEST-OF` -> `ACTIVE` + `SAMPLE-BEAM` endpoint, `CLAIM-C7-BESTOF-BEAM-API`): both request types gain `best_of`/`use_beam_search`/`length_penalty` + `to_beam_search_params`; `best_of>n` fans out via `SAMPLE-N` and returns the top-n by cumulative logprob (`SelectBestOf`); `use_beam_search` routes through the merged `BeamSearch` driver over the SYNC `LLMEngine` seam and returns `beam_width` beams as choices (endpoint output IDENTICAL to the direct driver call, gated completion + chat). Default (no best_of/beam) path byte-identical. HONEST FINDING: vLLM 0.26 exposes `use_beam_search` on the OpenAI server but has DROPPED `best_of` (only a vestigial `BatchChatCompletionRequest` field) — best_of follows classic OpenAI/V0 semantics gated on our own fan-out. Residuals: async-HTTP beam (driver is `LLMEngine&`-based, production server is AsyncLLM), streaming beam (rejected like upstream), C-ABI best_of/beam.** **Async/production beam LANDED 2026-07-28 (`SAMPLE-BEAM` async coverage, `CLAIM-C7-BEAM-ASYNC`): `BeamSearchAsync(AsyncLLM&, …)` mirrors vLLM `entrypoints/generate/beam_search/online.py:28-220`, drives the production AsyncLLM (the engine the HTTP server holds) per-beam single-token `generate` and calls the SAME merged `BeamSearchStep`/scoring (shared template driver; only the engine differs). `use_beam_search` on the server now RUNS instead of raising "requires the synchronous engine"; gated token-IDENTICAL to the sync `BeamSearch` (bw 1/2/3) + endpoint choices IDENTICAL over the AsyncLLM backend (completion + chat). Per-beam decodes driven SEQUENTIALLY (byte-identical to sync); concurrent per-beam stepping (online.py `asyncio.gather`) is a named residual. Remaining beam residuals: streaming beam, C-ABI beam, grammar-constrained beams.** | | 8 | `ROAD-V1-C8` | Tokenize/detokenize, unified streaming parsing and full metrics | [engine matrix](engine-matrix.md), [coverage views §§7,9](feature-matrix.md#7-structured-outputs--tool-calling), [metrics spec](specs/prometheus-metrics.md), [endpoints spec](specs/utility-endpoints.md) | **`SERVE-METRICS` + `SERVE-UTILITY-ENDPOINTS` LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8`, NOT pushed):** the oldest T0 debt — Prometheus `/metrics` (self-contained registry + text-0.0.4 exposition + always-on vLLM metric catalog, names/labels/buckets 1:1) — is CLOSED and gated by the vLLM scrape spec `EXPECTED_METRICS_V1` (RED-first substring gate, `test_prometheus_metrics` 4/4/81); plus `/tokenize`,`/detokenize`,`/ping`,`/server_info`,`/reset_prefix_cache` (additive/opt-in, schema-matched, `test_openai_api_server` 26/26/277). Inertness: opt-in, existing serving byte-identical. **`TOOLS-STREAMING-PARSER` core LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-PARSER`, NOT pushed):** the vLLM 0.26 unified streaming parser ENGINE (declarative `StreamingParserEngine`: token-ID scanner + prefix-buffering lexer + transition state machine + JSON-arg hold-back + drop-info) with qwen3/seed_oss/kimi_k2 configs + the unified registry, EXACT-gated event-for-event vs vLLM 0.26 (`test_streaming_parser_engine` 586/586, RED-first, goldens byte-reproduced from the pin). **Parser ASSEMBLY layer LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-ASSEMBLY`, NOT pushed):** the vLLM 0.26 `ParserEngine` (SemanticEvent -> streaming `DeltaMessage` + one-shot `ExtractedToolCallInformation`) with qwen3/seed_oss/kimi_k2 assembled parsers + `parser_manager` name->parser dispatch, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 9 scenarios, 1652/1652, RED-first 32 asserts, goldens byte-reproduced). **Serving-SSE dispatch swap LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-SERVING`, NOT pushed):** the OpenAI chat streaming path routes engine-backed `--tool-call-parser` names through `parser_manager get_parser_engine` and drives `parse_delta`/`parse`, EXACT chunk-for-chunk vs vLLM 0.26 `chat_completion_stream_generator` (`test_openai_serving_chat_stream` 9 scenarios, 210/210, RED-first 6 CHECKs, goldens byte-reproduced); OFF by default, legacy seam byte-identical (`test_openai_serving` 421/421 unchanged). **5 more engine CONFIG FAMILIES LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS`, NOT pushed):** minimax_m2/glm47_moe/deepseek_v4/deepseek_v32/nemotron_v3 ported as additive `ParserEngineConfig` builders + regex arg-converters + `Glm47MoeParser` name-`.strip()`, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 19 scenarios, 3510/3510, RED-first 2 asserts; engine-core 586/586 + serving-SSE 210/210 unchanged). **LAST 2 CONFIG FAMILIES gemma4 + inkling LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS-2`, NOT pushed) — vLLM tool-parser family parity CLOSED:** added 4 default-inert assembly-core virtual seams (`preprocess_feed`, virtual `events_to_delta`/`single_pass_parse`/`reset`/`extract_reasoning`, `args_wrapper_keys`) + `gemma4_config`/`inkling_config` (custom key:value + JSON-span arg carvers) + `Gemma4Parser` (channel-injection + `thought\n`-strip) / `InklingParser` (args-key unwrap + trailing-text flush), field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 26 scenarios, 4526/4526, adds a non-streaming parse() gate, RED-first for all 4 new seams; engine-core 586/586 + serving-SSE 210/210 byte-identical, seams inert). **LIVE PER-STEP METRIC WIRING LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-METRICS-WIRE`, NOT pushed) — the oldest T0 metrics debt fully RETIRED:** `/metrics` now serves LIVE values, not the primed schema. `EngineCoreOutputs.scheduler_stats` (new `Scheduler::make_stats()`, `scheduler.py:2399-2436`) + a stamped `timestamp`; `OutputProcessor::process_outputs` builds `IterationStats` (token counts, TTFT/ITL, finished-request breakdowns off `RequestState` timing, `stats.py:377-475`); the sync `LLMEngine::step()` folds both into the attached logger's `Record()` (`llm_engine.py:308-329`). Behavioural CPU gate `test_llm_engine.cpp` case 6 (44 assertions) drives the reference engine several steps: running/waiting gauges track the batch, prompt/generation counters == exact token counts, `request_success_total` counts finished reqs, TTFT/ITL/e2e/TPOT/iteration histograms observe the right sample counts; RED-first (14 asserts flip 0→correct with `Record` off). Additive/opt-in: null logger ⇒ byte-identical greedy stream; catalog gate `test_prometheus_metrics` 4/4/81 unchanged. **PER-REQUEST TIMING via EngineCoreEvents LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-RESPONSE-METRICS`, NOT pushed) — `SERVE-RESPONSE-METRICS` INVENTORIED→ACTIVE:** the scheduler records QUEUED/SCHEDULED/PREEMPTED `EngineCoreEvent`s 1:1 vLLM (`scheduler.py:2135/1003/1221`, gated on `log_stats_`), drained onto `EngineCoreOutput.events`; `OutputProcessor.update_from_events` folds them into `FinishedRequestStats.{queued,prefill,inference}_time` + `IterationStats.num_preempted_reqs`, so `vllm:request_{queue,prefill,inference}_time_seconds` + `vllm:num_preemptions_total` now carry real durations (the live-metrics wiring left them at 0). Gates: `test_scheduler` +1 (15 asserts, RED-first, real KV-exhaustion preemption) + `test_llm_engine` +1 (26 asserts, RED-first: 5 flip 0→positive; inference=prefill+decode, prefill≤inference≤e2e); no-logger path byte-identical. **CHAT-FORM `/tokenize` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-CHAT-TOKENIZE`, NOT pushed):** `/tokenize` accepts BOTH arms of the `TokenizeRequest` union — the chat form (`messages`+flags) renders through the SAME model chat template `create_chat_completion` uses (`chat_.prompt_fn()`), applies `check_generation_prompt` (→400), tokenizes with the chat-form `add_special_tokens` default False, returns `{count,max_model_len,tokens,token_strs?}` identically to vLLM 0.26; exact-gated tokens == render→Encode, RED-first (`test_openai_api_server` 27/27/337); raw-`prompt` form byte-identical. **JSON-SCHEMA ARG-TYPE COERCION LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-ARG-COERCION`, NOT pushed):** `_fix_arg_types`/`_streamable_string_keys`/`find_tool_properties` (parser_engine.py:227,269,365,348) ported over the already-ported `extract_types_from_schema`/`coerce_to_schema_type` helpers — `ParserTool` carries the function `parameters` schema (threaded from `serving_chat.cpp` `ToParserRequest`), so a request whose tools declare typed params has its assembled `tool_calls[].function.arguments` coerced to int/number/bool/string/array/null in BOTH streaming (`parse_delta`) and one-shot (`extract_tool_calls`/`parse`), 1:1 with vLLM (priority null>int>number>bool>object>array>string, uncoercible values left as-is); no-schema/absent-tools = identity byte-identical. Gate `test_parser_engine_assembly` 30 scenarios, 5038/5038 (scenarios 27-30: qwen3 typed-schema whole+char, schema-mismatch, kimi JSON-native `"5"`->int), RED-first 38 asserts (`"5"`-vs-`5`); engine-core 586/586 + serving-SSE 210/210 byte-identical. **`/tokenizer_info` (`SERVE-UTILITY-ENDPOINTS`) + `/abort_requests` (`SERVE-ADMIN`, INVENTORIED→ACTIVE) LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `GET /tokenizer_info` gated behind a `set_tokenizer_info_enabled` flag mirroring vLLM's `enable_tokenizer_info_endpoint` (off→route absent→404) surfaces the `tokenizer_config.json`-equivalent fields our byte-level/SentencePiece BPE tokenizer can GENUINELY back (`tokenizer_class`/`model_max_length`/`vocab_size`/`bos`/`eos`/`added_tokens_decoder`) and OMITS (never fabricates) the named gaps — the raw `chat_template` string (lives in the ChatPromptFn seam), the HF `init_kwargs`, and added-token `normalized`/`single_word`; `POST /abort_requests` (dev/rlhf admin router) parses `{request_ids}` and aborts the listed ids via an injected engine-abort callback (`AsyncLLM::abort`), empty→"abort all", returning `{"status":"aborted","aborted":N}` (malformed→400 `{"detail":...}`, failure→500 `{"error":...}`) — all shapes 1:1 vLLM. Both additive/opt-in (route only when backing attached), RED-first over a real socket (404 off → 200 on) + behaviour (abort tears down an in-flight AsyncLLM request); `test_openai_api_server` 31/31/379, existing conformance 23/23 + serving 40/40 byte-identical. **PRODUCTION `main.cpp` ENDPOINT WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary previously called NONE of the opt-in endpoint setters — a whole family of unit-tested endpoints was DARK in the actual server. A single shared seam `ConfigureUtilityEndpoints` (exercised by both `main.cpp` and the gate over a real socket) now lights, mirroring vLLM 0.26's per-endpoint default gating: `/tokenize`+`/detokenize` (on by default when a tokenizer exists), `/tokenizer_info` (new `--enable-tokenizer-info-endpoint`, mirrors `cli_args.py:140`), and `/abort_requests` (new `--enable-server-dev-mode`, mirrors `envs.VLLM_SERVER_DEV_MODE` at `api_server.py:238`) wired to the LIVE `AsyncLLM` with an exact before/after abort delta-count. `/metrics` + `/reset_prefix_cache` stay NAMED-BLOCKED (the AsyncLLM frontend exposes no live `PrometheusStatLogger` and no thread-safe prefix-cache reset RPC — attaching either would be a fabricated wiring). Core routes byte-identical; `test_openai_api_server` 32/32 (420, RED-first no-seam→404), conformance 23/23 + serving 40/40 unchanged; clean CPU `-Werror` incl. the `server` binary. RESIDUAL (honest): the chat/completion RESPONSE-BODY timing surface + CLI validation; AsyncLLM production-serving metric wiring (`/metrics` backing); config-gated metric families (spec-decode/kv-connector/mm/LoRA); `chat_template_kwargs`/`continue_final_message` full render passthrough; `/abort_requests` empty-list "abort ALL" enumeration; live `/reset_prefix_cache` backing | `PARTIAL` | response-body timing; AsyncLLM serving-path metric wiring + config-gated families; then DONE | | 9 | `ROAD-V1-C9` | Mechanical recurring upstream sync | [upstream sync](upstream-sync.md), [v0.25 audit](sync/2026-07-12-702f481.md), [porting inventory](porting-inventory.md) | v0.25.0 target `702f481` enumerated/classified: 145 post-pin commits, 94 inventory and 51 ignore, no unequivalent PORT-NOW runtime delta in the implemented T0 slice. The executable DGX oracle is validated/active at the target; the porting pin remains `e24d1b24` pending target goldens/behavior/model re-verification | `PARTIAL` | refresh exact performance denominators and target goldens/tests, then advance the parity pin | -| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | +| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **RPi5/A76 R2-R3 GREEN; `KERNEL-CPU-A76-Q8-DOT` SPIKED:** QEMU-built artifact is token/fixture exact on Pi; portable Q8 dot is 20.10% of user cycles and exact-order C++ SDOT vs scheduled AAPCS64 is the active gate. **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | | 10a | `ROAD-V1-D1-CUDA` | **CUDA-arch derive-and-ship + community testing matrix (user-directed 2026-07-27):** the arch-additivity framework already makes a new CUDA device a near-verbatim add, so DERIVE-AND-SHIP the fast-path kernels rather than scope-and-wait — port 1:1 from vLLM (Ampere FA2/Marlin/AllSpark/scaled-mm-C2x; Hopper FA3/C3x-Sm90/Machete/MoE; datacenter-Blackwell C3x-Sm100a-tcgen05/MLA/fp4), and the fp16/non-tensor-core bodies from llama.cpp for the arches vLLM DROPS (Pascal/Volta/Turing = MORE than vLLM), build-verify (compile + `cuobjdump` SASS), and SHIP LABELED with a 3-state signal: RUNTIME-VERIFIED / DERIVED+BUILD-VERIFIED (testing-welcome) / NOT-YET-BUILDABLE (a build is never a runtime claim). Runtime-gate boards: **AGX Orin (sm_87, reachable)** + **NVIDIA Thor (Blackwell, reachable, SM to confirm on-box)** — the first two non-GB10 runtime proofs, spanning Ampere + Blackwell. DeepGEMM (runtime JIT) is the one honest NOT-YET-BUILDABLE. | [backend matrix](backend-matrix.md), [Ampere](specs/cuda-arch-ampere-fastpath.md), [Hopper+DC](specs/cuda-arch-datacenter-fastpath.md), [breadth](specs/cuda-arch-breadth-fp16.md) | 3 committed spikes (2026-07-27); backend rows moved INVENTORIED/build-only -> SPIKE; **FIRST NON-GB10 RUNTIME PROOF: Thor sm_110 portable bf16 path RUNTIME-VERIFIED (token-exact Llama-3.2-1B, `CLAIM-CUDA-SM110-RUNTIME`)**; **FIRST DERIVE-AND-SHIP BRICK: WA-1 FA2 Ampere enablement DERIVED+BUILD-VERIFIED (fa2 cell widened to 8.x; sm_87/sm_80 FA2 cubins, sm_121a SACRED 16/16 unchanged, no Ampere board ran it)**; **Turing sm_75 W1 BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-TURING-SM75`): the bf16-WMMA TU guard landed, single-arch `75` `-Werror` 0-warn + `cuobjdump` real `sm_75` cubin, sm_121a byte-identical — DERIVED+BUILD-VERIFIED (testing-welcome), no board ran it**; **DC1 datacenter-Blackwell sm_100a NVFP4 tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-NVFP4`): faithful 1:1 port of vLLM `Fp4GemmSm100` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05) as a NEW dedicated `cutlass-nvfp4-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror`-equiv 0-warn + `cuobjdump` real `sm_100a` cubin, RED (HEAD 100a fp4 DISABLED), sm_121a neutral — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; native `fp4-mma` stays sm_12x-only (mma.sync≠tcgen05)**; **DC2 Hopper sm_90a CUTLASS C3x FP8 scaled-mm BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM90-C3X`): faithful 1:1 port of vLLM `cutlass_3x_gemm_sm90_fp8` (`ArchTag=Sm90`+`KernelTmaWarpSpecialized*FP8FastAccum`→wgmma/TMA) as a NEW dedicated `scaledmm-c3x-sm90` FEATURE-TABLE cell (90a-only), single-arch `90a` `-Werror` 0-warn + `cuobjdump` real `sm_90a` cubin (ptxas C7510 names `wgmma.mma_async`), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no Hopper board ran it; sm90 int8/blockwise + sm100 C3x legs are residual bricks**; **DC3 datacenter-Blackwell sm_100a CUTLASS C3x FP8 scaled-mm tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-C3X`): the intersection of DC1's tcgen05 arch + DC2's C3x fp8 kernel — a faithful 1:1 port of vLLM `cutlass_3x_gemm_sm100_fp8` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05, `sm100_fp8_config_{default,M256,M64}`, 2SM `<_2,_2,_1>` default) as a NEW dedicated `scaledmm-c3x-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror all-warnings` 0-warn EXIT=0 + `cuobjdump` real `sm_100a` cubin (`Sm100TmaUmma`/`SM100_MMA_F8F6F4_2x1SM`/`TMEM` symbols, `LDTM`/`tmem` SASS), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8`/`cutlass-nvfp4`/`cutlass-nvfp4-sm100` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; sm100 int8/blockwise + MoE/MXFP4/MLA legs are residual bricks**; Volta/Pascal need CUDA<13 | ACTIVE | **SECOND NON-GB10 RUNTIME PROOF: Orin sm_87 portable bf16 SYNC path RUNTIME-VERIFIED 2026-07-28 (`CLAIM-CUDA-ORIN-SM87-RUNTIME`)** — built portable-only for sm_87 on a real Jetson AGX Orin (CUDA-13 container BLOCKED by the 12.6 driver → l4t-jetpack r36.4.0 nvcc 12.6 + g++-13; all fast paths EMPTY for [87]), and `test_llama_paged_engine` (Llama-3.2-1B bf16) is **13/16 strict token-exact vs the vLLM 0.25.0 oracle, 16/16 near-tie gate, 0 divergent** on the sm_87 GPU (exceeds Thor's 12/16), + test_cuda_backend/test_cuda_ops 461 assertions of real on-device kernels; ONE honest sm_87 bug found — the DEFAULT async runner crashes (`cudaFree illegal memory access`), SYNC path (`VT_ASYNC_RUNNER=0`) is the verified one (unblock item). Residual Orin W-plan: fix async runner -> GGUF leg -> FA2 (needs cutlass) -> Marlin-int4 -> vs llama.cpp; Thor fp8/fp4 fast paths once SSH-reachable | | 10b | `ROAD-V1-D1-GDN-AOT` | **Per-arch Triton-AOT GDN packed-decode cubins (user-directed 2026-07-28):** the vendored Triton-AOT GDN fast-path cubins — the MEASURED codegen-win decode (Triton REG:205/0-spill vs hand-CUDA REG:255+STACK:48 spills) plus the delta_h/chunk_o/kkt/tril/wu FLA set — existed for **`sm_121a` ONLY**, so GDN-hybrid models (Qwen3.6) ran GDN decode correct but on the SPILLING hand kernel on every other arch (all cross-family arch builds ship `-DVLLM_CPP_TRITON=OFF`). Regenerate the full GDN AOT set per-arch via the sanctioned `-DVLLM_CPP_TRITON_REGEN=ON` pipeline and vendor each `sm_XX/` tree so a `-DVLLM_CPP_TRITON=ON` build selects the non-spilling FLA path additively (build-time cubin selection = `sm_${VLLM_CPP_CUDA_ARCHITECTURES}`, no code change). | [kernel matrix](kernel-matrix.md), [backend matrix](backend-matrix.md) | [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `DONE (DERIVED+BUILD-VERIFIED)` | **LANDED 2026-07-28 (`CLAIM-TRITON-AOT-PER-ARCH`, base `308c312a`):** full GDN AOT set regenerated + vendored for `sm_80/86/89/90a/100a` on dgx GB10 (Triton 3.6.0 / ptxas 12.8, cross-compiled — no target board needed; 57 artifacts + MANIFEST per arch matching the `sm_121a` fileset), `cuobjdump` proves real per-target SASS (`sm=80/86/89/90/100`, decode REG 209–217/0-spill, under the hand-CUDA spill floor); builder-path configure selects + integrity-verifies each tree; `check-triton-aot-drift.sh` rc=0 across all six trees; `sm_121a` byte-untouched (SACRED 27B/35B gate structurally unchanged). **HONEST: DERIVED+BUILD-VERIFIED — no non-`sm_121` board runs a GDN model here, so decode parity on these arches is build-verified, NOT runtime-measured.** Residual: runtime GDN-model verification on Ampere/Ada/Hopper/DC-Blackwell silicon when reachable | | 11 | `ROAD-V1-D2` | Tensor/multi-GPU and MoE sequence parallelism | [engine matrix](engine-matrix.md), [coverage view §3](feature-matrix.md#3-parallelism--scale-out) | TP spec written; v0.25.0 non-DP MoE sequence-parallel path is inventoried as `PAR-SEQUENCE-MOE` but unspiked | `READY` | acquire 2-GPU target and claim Phase 0 mock/ABI; its execution trace determines whether sequence parallel is part of the first performance slice | diff --git a/.agents/specs/cpu-a76-q8-dot.md b/.agents/specs/cpu-a76-q8-dot.md new file mode 100644 index 000000000..b38153a98 --- /dev/null +++ b/.agents/specs/cpu-a76-q8-dot.md @@ -0,0 +1,161 @@ +# KERNEL-CPU-A76-Q8-DOT spike + +Date: 2026-08-06 + +Row: `KERNEL-CPU-A76-Q8-DOT` + +Claim: `CLAIM-KERNEL-CPU-A76-Q8-DOT`, stacked branch +`row/KERNEL-CPU-A76-Q8-DOT` + +State: `SPIKE` + +## Scope + +Optimize the Q8_0 weight by Q8_0 activation dot used by +`vt::MatmulBTQuant` on Raspberry Pi 5's Cortex-A76. The row owns a portable +fallback, a compiler-generated Arm DotProd/SDOT implementation, a scheduled +AAPCS64 implementation, runtime dispatch, same-binary selection, focused tests, +disassembly evidence and recursive Qwen3.5-2B measurements. + +In scope is `nrc == 1` and K divisible by the 32-element Q8_0 block. The +existing i8mm `nrc == 2` path, other quantization formats, BF16 GEMM, model +semantics, the threadpool and non-A76 scheduling are out of scope. DotProd +cores other than A76 may use the compiler implementation after correctness is +proven, but the assembly schedule is selected only for Cortex-A76. + +## Upstream chain + +- Pinned vLLM `555967922` supplies Qwen3.5 semantics, not this low-level CPU + kernel. Its CPU platform entry is `vllm/platforms/cpu.py:42-125` and compiled + ops begin at `csrc/cpu/torch_bindings.cpp:123-139`. +- The project llama.cpp pin `237ad9b96` owns the performance reference: + `ggml/src/ggml-cpu/quants.c:400` is the portable Q8_0 dot and + `ggml/src/ggml-cpu/arch/arm/quants.c:1076-1160` is its Arm NEON/DotProd + implementation. The latter is design evidence, not a license to change the + project's reduction contract. +- Runtime truth is the R3 physical-Pi trace: 241K `cycles:u` samples, zero + loss, with portable `VecDotQ8_0Q8_0` at 20.10% and `F16ToF32` at 4.87%. + The Pi reports ASIMDDP/DotProd and no i8mm, so the scalar dot is genuinely + selected while the existing Arm fast tier is inert. + +## Our baseline + +- `src/vt/cpu/cpu_quant_dot.cpp:88-115` executes one scalar integer dot per + block, converts two binary16 scales through `F16ToF32`, and accumulates one + float contribution per block in strict order. +- `src/vt/cpu/cpu_quant_gemm.cpp:72-94` calls that function for every output + element. `src/vt/cpu/cpu_quant_dot_arm.cpp:538-548` only offers i8mm MMLA, + which the target lacks. +- `tests/vt/test_ops_quant_dot.cpp:559-585` ports llama.cpp's Q8 dot error case; + the surrounding cases gate fixed reduction, ragged K, GEMM NMSE and + thread-count determinism. +- `examples/cpu_kernel_bench/main.cpp` supplies exact Q8 fixtures, cache modes, + 1/2/4-core affinity and grouped PMU counters. The portable M=1/N=3072/K=2048 + denominator is 1,554,115 ns on one core and 742,585 ns on four, checksum + `0xd6aec014c0050fda`. + +## Port map + +| Source | Local destination | Decision | +|---|---|---| +| llama portable `quants.c:400` | existing `cpu_quant_dot.cpp` | permanent fallback and exact oracle | +| llama Arm DotProd structure | new `cpu_quant_dot_sdot.cpp` | ACLE `vdotq_s32`, exact per-block float order | +| measured A76 schedule | new `cpu_quant_dot_a76.S` | original implementation, AAPCS64, no stack spill in core loop | +| Linux HWCAP/MIDR selection | new C++ TU plus `include/vt/quant.h` | ASIMDDP gate; A76-only assembly; env A/B control | +| quant GEMM dispatch | `cpu_quant_dot.cpp` | selected function replaces only Q8_0 row | +| benchmark variants | `examples/cpu_kernel_bench/main.cpp` | `portable`, `sdot`, `a76-asm`, `auto` | + +The optimized implementations return exactly the portable function signature. +They perform both 16-byte SDOTs per Q8 block, horizontally add the four i32 +lanes, then apply scales and update the scalar f32 accumulator in original +block order. Integer reassociation is exact; float block order is not changed. + +## Tests to port + +- Retain the upstream-derived `test-quantize-fns` Q8 dot error case already at + `tests/vt/test_ops_quant_dot.cpp:559`. +- Extend its Q8 arm to invoke portable, C++ SDOT and assembly directly on + random, all-zero, signed-extreme and scale-edge blocks; require bit identity + for finite normal scale cases and explicit IEEE agreement for special scales. +- Gate K={32,64,96,2048}, unaligned-but-valid block bases, `nrc != 1`, ragged K, + and null/unsupported dispatch behavior. +- Add feature-selection cases: no DotProd -> portable, DotProd non-A76 -> C++, + A76+DotProd -> assembly, and environment overrides for every variant. +- Keep `MatmulBTQuant` bit-exact across 1/2/4 threads and run ASan/UBSan on the + generic build. An x86 build must link only the stubs and retain current output. + +No vLLM Python test specifies instruction scheduling. Existing Qwen3.5 GGUF +model tests and the pinned 16-token current-engine golden remain the semantic +e2e test. + +## Gates + +Correctness: + +1. x86/generic AArch64 build and current quant suite remain green. +2. QEMU executes every explicit variant and preserves exact fixture checksums. +3. Physical Pi variants preserve both portable checksums and the 16-token x86 + golden. No threshold may be widened to admit a faster answer. + +Compiler-gap and ABI: + +1. Save `objdump -drC` for portable, C++ SDOT and assembly symbols. The C++ + object must prove SDOT issuance; assembly must prove the intended load/SDOT, + scale-convert and accumulation schedule, no core-loop stack traffic, only + caller-saved clobbers, and no instruction above Armv8.2-A+DotProd+FP16. +2. Compare C++ SDOT to assembly in the same binary. Assembly is accepted only + if physical-Pi wall time and cycles improve outside run noise; beating only + the scalar baseline is insufficient. + +Performance: + +1. Use randomized interleaved portable/C++/assembly fixture trials, at least + seven samples per arm, on an idle unthrottled Pi at recorded frequency. +2. Record median/min wall time, cycles, instructions, IPC, frontend/backend + stalls and last-level traffic for M=1 and M=128 at one and four threads. +3. Run at least three interleaved 64-token Qwen arms. Assembly must improve the + enclosing model or be kept opt-in with its kernel-only win recorded. Exact + output tokens, TTFT/TPOT/ITL, load time, RSS, temperature and throttling are + recorded. QEMU timings are never evidence. + +The final CPU-backend floor against same-file llama.cpp is a later campaign +gate; this row only closes the proven Q8 assembly/compiler gap. + +## Dependencies + +- `BACKEND-CPU` R2-R3 commit `16c848326`, the pinned Q8_K_XL model and golden. +- Local Docker buildx builder `pf-arm`; QEMU compiles/tests, never benchmarks. +- Disposable execution directory `rich@rpi5fan.lan:~/vllm-cpp-assembly`. +- Linux `getauxval(AT_HWCAP)` and Cortex-A76 implementer/part detection. +- No new library, generated binary or external source dependency. Any borrowed + code would require a separate license/attribution review; the assembly is + written for this row from the stated instruction/dataflow contract. + +## Work breakdown + +| W | Work | Exit | +|---|---|---| +| W0 | spike, row and claim | record checkers green, committed before code | +| W1 | selection seam + explicit benchmark variants | x86 fallback and QEMU smoke green | +| W2 | exact-order C++ SDOT | disassembly contains SDOT; fixture/model exact | +| W3 | scheduled AAPCS64 kernel | ABI/disassembly checks and direct tests green | +| W4 | interleaved Pi PMU A/B | assembly beats C++ in cycles and wall time | +| W5 | recursive Qwen A/B and checkpoint | no output/enclosing regression; docs current | + +## Risks and decisions + +- The compiler may already schedule the intrinsic loop optimally. In that case + this assembly candidate is negative and the row remains open or selects a + different proven compiler gap; an assembly file is not success by itself. +- Q8 blocks are 34 bytes, so scale and payload streams are not naturally + 16-byte aligned. Loads must remain unaligned-safe, and speculative reads may + not cross the allocated final block. +- Hardware FP16 conversion must match `F16ToF32`, including signed zero, + infinities and NaNs. If payload handling differs, retain software conversion + for specials or dispatch those inputs to portable code. +- Unrolling may expose memory-level parallelism but increases register and I- + cache pressure. PMU plus recursive model A/B, not instruction count alone, + decides the retained schedule. +- The R3 model is BF16-GEMM dominated. A clear dot-kernel win may be diluted; + it is still accepted as the requested demonstrated assembly win only when no + enclosing regression exists, and remains opt-in until model wall time moves. diff --git a/.agents/state.md b/.agents/state.md index 6af35f989..70143dfe9 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39364,3 +39364,19 @@ R4/R5 will compare an exact-order C++ SDOT intrinsic against a scheduled AAPCS64 implementation in one QEMU-built binary, using the portable checksums, PMU fixture and full-model run as recursive gates. No speedup or llama.cpp parity is claimed at this checkpoint. + +## 2026-08-06T17:50 - KERNEL-CPU-A76-Q8-DOT spiked + + +R3's zero-loss physical-Pi trace selects the portable Q8_0 x Q8_0 dot as the +first assembly candidate: it consumes 20.10% of Qwen3.5-2B user cycles on a +Cortex-A76 with DotProd but no i8mm. New row `KERNEL-CPU-A76-Q8-DOT` and spike +`.agents/specs/cpu-a76-q8-dot.md` bind the comparison to three same-binary arms: +portable scalar, exact-order compiler SDOT, and scheduled AAPCS64 assembly. + +The assembly arm is not accepted merely for using SDOT or beating scalar. It +must beat the C++ intrinsic in physical-Pi cycles and wall time with identical +operation checksums and model tokens, a proven ABI/disassembly contract, and no +enclosing Qwen regression. QEMU remains build/smoke-only and source will not be +compiled on the Pi. No optimized implementation or speed claim exists at this +spike checkpoint. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 11cc717e2..62112bf9d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -169,6 +169,10 @@ dot. These numbers bind the portable denominator only. SDOT/assembly A/B, three interleaved full-model repetitions, peak memory and same-file llama.cpp comparison remain `PENDING`. +The selected follow-on is `KERNEL-CPU-A76-Q8-DOT`. Its binding performance +gate compares portable, exact-order compiler SDOT and scheduled AAPCS64 in one +QEMU-built binary on the physical Pi. No assembly speedup is claimed at spike. + Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index f751d4bd7..cf9d69378 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; QEMU-built A76 correctness/PMU baseline, SDOT gating) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; A76 baseline green, Q8 SDOT/assembly spiked) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index f28c2738e..f4990038b 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi R2-R3: QEMU-built artifact is 16/16 x86-token exact and fixture-exact; unthrottled portable baseline captured, with Q8 dot 20.10% of user cycles. SDOT/assembly A/B and Pi llama.cpp floor remain open. Concurrency open | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi R2-R3 is 16/16 x86-token and fixture exact; Q8 dot is 20.10% of user cycles. `KERNEL-CPU-A76-Q8-DOT` is `SPIKE`: exact-order C++ SDOT vs AAPCS64 assembly A/B next; Pi llama.cpp floor open. Concurrency open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 5e5da2910..87ec25279 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -130,9 +130,14 @@ # the DGX GB10: 2/2 cases · 92401 assertions vs the CPU oracle + f64 dequant, # memcheck 0, RED-first proven. `ACTIVE`, `CLAIM-CUDA-KEEPQUANT-GEMM`, spec # specs/deepseek-v4-flash.md §W8.) + # 46 since 2026-08-06: +`KERNEL-CPU-A76-Q8-DOT`, a separately gateable + # Cortex-A76 Q8_0 x Q8_0 DotProd/assembly family. The physical-Pi trace + # proves the portable dot is reached at 20.10% of Qwen3.5-2B user cycles; + # the row owns exact-order C++ SDOT vs scheduled AAPCS64, independent of + # the broad CPU-backend row. # Inventory size, bumped for a genuinely new family — never to make a failing # state transition pass. - "KERNEL": (AGENTS / "kernel-matrix.md", 45), + "KERNEL": (AGENTS / "kernel-matrix.md", 46), # 56 since 2026-07-22: +`BACKEND-ACCEL-PROVIDER` (the acceleration-provider seam # itself, which is a cross-backend platform concern rather than a platform). # 57 since 2026-07-22: +`BACKEND-SEAM-AUDIT` (the accelerator-seam AUDIT — does From 9044c2a7d828295bfed834f876e4f255a4d4a936 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 20:19:50 +0100 Subject: [PATCH 6/7] Optimize Cortex-A76 Q8 dot with scheduled SDOT assembly Add runtime-gated exact-order compiler and AAPCS64 variants, QEMU gates, and bind the physical-Pi compiler-gap result. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 2 +- .agents/backend-matrix.md | 2 +- .agents/coordination.md | 14 -- .agents/feature-matrix.md | 2 +- .agents/kernel-matrix.md | 2 +- .agents/parity-ledger.md | 1 + .agents/roadmap_v1.md | 2 +- .agents/specs/cpu-a76-q8-dot.md | 68 +++++--- .../specs/rpi5-cortex-a76-cpu-optimization.md | 28 +++- .agents/state.md | 58 +++++++ CMakeLists.txt | 12 ++ docker/Dockerfile.arm64 | 19 ++- docs/BENCHMARKS.md | 39 +++-- docs/ENVIRONMENT.md | 1 + docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- .../rpi5-a76-q8-dot-20260806.md | 145 ++++++++++++++++ examples/cpu_kernel_bench/main.cpp | 18 +- include/vt/quant.h | 11 ++ src/vt/cpu/cpu_quant_dot.cpp | 3 +- src/vt/cpu/cpu_quant_dot_a76.S | 94 +++++++++++ src/vt/cpu/cpu_quant_dot_sdot.cpp | 156 ++++++++++++++++++ tests/vt/test_ops_quant_dot.cpp | 74 +++++++++ 23 files changed, 685 insertions(+), 70 deletions(-) create mode 100644 docs/bench-evidence/rpi5-a76-q8-dot-20260806.md create mode 100644 src/vt/cpu/cpu_quant_dot_a76.S create mode 100644 src/vt/cpu/cpu_quant_dot_sdot.cpp diff --git a/.agents/NOW.md b/.agents/NOW.md index 9567ff8bf..68578bfe4 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| RPi5 A76 CPU | **R3 BASELINE GREEN:** QEMU-built artifact, 16/16 x86-token exact, PMU trace | Portable Q8 dot is 20.1% of user cycles; prove C++ SDOT/compiler gap, then A76 assembly A/B | +| RPi5 A76 CPU | **R5 ASSEMBLY GREEN:** output-exact AAPCS64 beats compiler SDOT 3.66-5.08%; Qwen TTFT/E2E improve | W6: M1/T4 −2.43%, BF16 GEMM, memory/concurrency and llama.cpp floor | | MXFP4 parity | **TERMINAL:** c1 1.020x pass; c2-c8 0.962-0.969 GPU-intrinsic; final cap/glue levers exhausted | Record; branch unmerged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index f137ea0cf..08d786c3a 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -223,7 +223,7 @@ on every listed target. | `BACKEND-ABI-VT` | Backend registry, device/queue, capture, drop-in adapter ABI | platform contract `vllm/platforms/interface.py:67-229`; stable stream `csrc/libtorch_stable/torch_utils.h:76-82`; raw Marlin boundary `csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu:344-356` | queue ID [device.h:9](../include/vt/device.h#L9), explicit resources [backend.h:53](../include/vt/backend.h#L53), scalar/layout ABI [ops.h:13](../include/vt/ops.h#L13), CUDA workspace/raw probe [cuda_dropin.cu:104](../src/vt/cuda/cuda_dropin.cu#L104) | [ported ABI tests](../tests/vt/test_dropin_abi.cpp#L86); CPU 94/94 + repair-focused 1/1; GCC13 repair at [test_dropin_abi.cpp:245](../tests/vt/test_dropin_abi.cpp#L245); exact sm_121a all-target build 100%, focused CUDA/ABI CTest 2/2, sanitizer 9/9 + 196/196 with 0 errors/leaks, 35B/27B gates 2/2 (`1141b79`, evidence manifest `4adbe952…601`) | [drop-in ABI](specs/dropin-kernel-abi.md) | `ANCHOR-BACKFILL` - sm_80/sm_90a cross-build and unchanged-trace/model A/B-memory proof remain; common scalar forwarder and legacy backend-shim migration remain; no production family migrated | CLAIM-BACKEND-ABI-W0-GPU-1 | | `BACKEND-ATTN-REGISTRY` | Attention-backend registry + platform-driven priority selection (extensibility **item 4** — the second portability seam, now REALIZED). Backends SELF-REGISTER per `(DeviceType, name)`; `Platform::get_attn_backend_priority()` advertises a capability-ordered name list; the selector returns the first REGISTERED name. Selection is DATA (register + priority slot), not an inline code edit. The concrete attention KERNEL stays at the vt:: op-table (`vt::PagedAttention`→`GetOp`), already device-additive | `vllm/v1/attention/backends/registry.py` (self-registration), `vllm/platforms/cuda.py:361-470` (`get_valid_backends`/`get_attn_backend_cls`) + `:84-166` (`_get_backend_priorities`, non-MLA major-10 vs else), `cpu.py:75-87` (CPU_ATTN) @ pin `e24d1b24` | registry+selector [registry.h:44](../include/vllm/v1/attention/registry.h#L44), [registry.cpp:60](../src/vllm/v1/attention/registry.cpp#L60); priority [interface.h:92](../include/vllm/platforms/interface.h#L92), CUDA major-10/else [cuda.cpp:44](../src/vllm/platforms/cuda.cpp#L44), CPU [cpu.cpp:29](../src/vllm/platforms/cpu.cpp#L29); FLASH_ATTN self-reg [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), GDN [gdn_attn.cpp:153](../src/vllm/v1/attention/backends/gdn_attn.cpp#L153) | [registry tests](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L61) (self-register + Make/throw + CUDA/CPU priority order + first-registered walk + override; 8 cases/25 assertions) + [platform priority](../tests/vllm/platforms/test_platform.cpp#L53); clean CPU `-Werror` + full CPU CTest green; behavior-preserving ⇒ same FA2 attention selected (**DGX-CONFIRMED @ `2c732e7`: 27B 235/235 + 35B 315/315 token-exact, FA2 sm_121a, memcheck 0/315**) | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) (item 4) | `ANCHOR-BACKFILL` — item-4 seam LANDED CPU + DGX-CONFIRMED; FLASHINFER/TRITON_ATTN/FLEX/TURBOQUANT are named in the priority (fidelity) but unimplemented ⇒ walk falls through to FLASH_ATTN (the behavior-preserving outcome); MLA-branch priorities deferred until an MLA model ports | CLAIM-ATTN-REGISTRY-1 | | `BACKEND-PLATFORM` | Platform capability/memory-model seam (`is_cuda`/`is_cpu`, `is_unified_memory`, `has_device_capability`, `supported_dtypes`, `residency_policy`, `supports_graph_capture`; `get_attn_backend_priority` REALIZED by `BACKEND-ATTN-REGISTRY` item 4; **+ the S3 capability predicates `supports_fp8`/`cutlass_fp4_supported`/`opaque_attention_op`/`is_integrated_gpu`/`support_static_graph_mode`/`is_device_capability_family`, mirroring `interface.py:914,933,977,1058,441-476` + `nvfp4_utils.py:56`, added by `CLAIM-BACKEND-SEAM-S3-1` and consumed by the fp4/fp8/graph gates in `qwen3_5.cpp`**); self-registered per `DeviceType` via `CurrentPlatform()` | `vllm/platforms/interface.py:134-229` (`class Platform`), `:409-439` (get/has_device_capability), `:181-187` (supported_dtypes) @ pin `e24d1b24` | interface [interface.h:56](../include/vllm/platforms/interface.h#L56), registry/CurrentPlatform [platform.cpp:57](../src/vllm/platforms/platform.cpp#L57), CPU [cpu.cpp:11](../src/vllm/platforms/cpu.cpp#L11), CUDA [cuda.cpp:14](../src/vllm/platforms/cuda.cpp#L14); composes `vt::Backend` [backend.h:22](../include/vt/backend.h#L22) | [platform tests](../tests/vllm/platforms/test_platform.cpp#L28) (registration + CPU capabilities + has_device_capability), clean CPU `-Werror` + full CPU CTest; 7 memory-model/residency sites migrated (`runner.cpp`, `model_registry.cpp`, `qwen3_5.cpp`) — **per-tensor sites key on `GetPlatform(.device.type).is_cuda()` (the OBJECT's device), NOT the process-global `CurrentPlatform()`, which is accelerator-first and would mis-route a CPU queue/tensor on a GPU box (regression fixed 2026-07-18; red DGX CPU tests `test_platform`/`test_qwen27_dense_forward`); `CurrentPlatform()` is reserved for process-level "which accelerator" questions only**. DGX behavior-preserving model gates 27B 235/235 + 35B 315/315 CONFIRMED @ (CUDA -Werror-clean after a GCC13 dangling-pointer FP fix). **Item-2 residency_policy() CONSUMPTION LANDED (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`):** the host-free / load-stream / DevicePool-cap decisions in `qwen3_5.cpp` now READ `GetPlatform(.device.type).residency_policy()` (per-device) instead of an inline `device.type`/env gate — host-free `qwen3_5.cpp:BuildMoeMarlinResident` via `ShouldReleaseHostWeights`, load-stream `Qwen3_5Model::PrepareMarlinResident` via `ShouldInterleaveLoadStream` (both `interface.h`), DevicePool soft cap via `residency_policy().device_pool_cap_bytes` (`DBuf`/`DevicePool`). `CudaPlatform::residency_policy().release_host_weights_after_upload` flipped **false→true** (now CONSUMED ⇒ reproduces today's GB10 host-free-after-Marlin-build EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH gate; `VT_MOE_HOST_FREE`/`VT_MOE_LOADSTREAM` env stay as overrides. Consumption unit-tested `test_platform.cpp` (7 cases/43 assertions). A new (discrete) GPU sets `residency_policy()` values ⇒ ZERO model edit. **DGX gate PASSED @ `62fc0e0` (production flags CUTLASS sm120a + FA2 sm_121a + Triton AOT, one flock): clean CUDA `-Werror` 0 warn, 27B 235/235 + 35B 315/315 token-exact, 35B VmHWM ≈ 4.0 GiB (load-stream win preserved), memcheck 0 errors.** | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) | `ANCHOR-BACKFILL` — item-1 extraction DONE + DGX-CONFIRMED; item-2 residency consumption LANDED + DGX-PROVEN (`CLAIM-BACKEND-PLATFORM-2`); kernel-shape dispatch branches deliberately left for items 4/5 | CLAIM-BACKEND-PLATFORM-1 / CLAIM-BACKEND-PLATFORM-2 | -| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06 (PR #65):** R0 pins the exact Q8_K_XL hash, four-core A76 DotProd/no-i8mm target, recursive gates and llama.cpp floor. R1 [PMU harness](../examples/cpu_kernel_bench/main.cpp) is CPU-gated. **R2-R3 GREEN:** QEMU-built AArch64 artifacts pass smoke and execute on the Pi; Qwen3.5-2B is exact 16/16 against the x86 token golden, four operation fixtures retain exact checksums, and unthrottled PMU baselines are captured. A zero-loss model trace ranks portable Q8 dot at 20.10% of user cycles (BF16 GEMM 57.76%); the Pi has DotProd but no i8mm, so C++ SDOT and AAPCS64 assembly A/B are the active next gate. No optimized or llama.cpp-parity result is claimed; no 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | +| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06:** R0-R3 pin the exact Q8_K_XL vehicle, local buildx/QEMU build and physical-Pi correctness/PMU baseline. **R4-R5 GREEN:** [KERNEL-CPU-A76-Q8-DOT](kernel-matrix.md) adds exact-order compiler SDOT and a scheduled AAPCS64 leaf; Pi operation and 64-token model outputs are exact. Assembly beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4, lowers Qwen TTFT 1.55% and E2E 0.13%, so A76+DotProd auto-selects it. M1/T4 regresses 2.43%; thread partition, peak memory, concurrency and same-file Pi llama.cpp floor stay open. No 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | | `BACKEND-CPU-ZEN` | AMD Zen CPU with ZenDNN/zentorch dispatch and weight prepack | `platforms/zen_cpu.py:12-32`; detection `platforms/__init__.py:153-192`; `tests/test_zen_cpu_platform_detection.py:8-37` | generic CPU backend only; no Zen-specific dispatch | - | [CUDA inventory](specs/cuda-architecture-inventory.md); leaf spike required | `INVENTORIED` | - | | `BACKEND-ROCM` | AMD ROCm/HIP | `platforms/__init__.py:110-128`, `platforms/rocm.py:43-125`, ROCm ops rooted at `csrc/rocm/moe_q_gemm_rdna3.cu:1` | Lands through the realized seams: a `platforms/rocm.cpp` Platform + a `kROCM` attention TU self-registering via the [attn-registry seam](../include/vllm/v1/attention/registry.h#L44) + a `rocm.cpp` `get_attn_backend_priority()` slot (mirror `rocm.py`'s ROCM_ATTN/AITER ordering); ZERO selector/model/runner edit | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-XPU` | Intel XPU loyal port; gating exploration E4 ([backends.md](backends.md) kernel sourcing: SYCL-vs-Level-Zero call + whether upstream XPU attention contracts translate 1:1; explore when T2 scheduling begins) | `platforms/__init__.py:131-150`, `platforms/xpu.py:103-125` | enum slot only [device.h:11](../include/vt/device.h#L11) | [unavailable-backend test](../tests/vt/test_backend.cpp#L37); **SPIKE-ONLY verdict, MEASURED 2026-07-22: HW-BLOCKED — no Intel GPU on any box we own, and no acquisition path recorded.** Compounding finding: the loyalty target is INCOMPLETE upstream — `vllm/platforms/xpu.py:11-13` imports the EXTERNAL `vllm_xpu_kernels._C`/`._moe_C`/`._xpu_C`, and a search of the pinned tree for `*.sycl`/`*.dp.cpp` returns NOTHING, so there is no in-tree SYCL kernel source to mirror 1:1. What stays doable with zero Intel hardware: the attention-selector/priority POLICY port (pure data, unit-testable), oneAPI DPC++ compile coverage, and SYCL kernels executed on the oneAPI OpenCL CPU device for unit-level numerics. **No e2e or performance gate is proposed, because none can be run.** No XPU backend code exists | [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `SPIKE` | `CLAIM-BACKEND-FANOUT-1` | diff --git a/.agents/coordination.md b/.agents/coordination.md index 788c454cb..64982d198 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,19 +118,6 @@ without the selected contention proof for their entire run are discarded. ## Active claims -**Cortex-A76 Q8 SDOT/assembly (`KERNEL-CPU-A76-Q8-DOT`, 2026-08-06, -`CLAIM-KERNEL-CPU-A76-Q8-DOT`).** Codex (GPT-5), stacked local branch -`row/KERNEL-CPU-A76-Q8-DOT` in isolated worktree -`/tmp/vllm-cpp-row-BACKEND-CPU`, base `BACKEND-CPU` R2-R3 commit `16c848326`. -Owns only the new A76 Q8 dot C++/assembly TUs, their `include/vt/quant.h` and -CMake wiring, the Q8-specific portions of `cpu_quant_dot.cpp`, -`test_ops_quant_dot.cpp` and `examples/cpu_kernel_bench/main.cpp`, plus this -claim, `KERNEL-CPU-A76-Q8-DOT`, its spike and required checkpoint surfaces. -Builds ARM64 only through local buildx/QEMU; deploys exported artifacts into -the user-authorized disposable `rich@rpi5fan.lan:~/vllm-cpp-assembly` for -uncontended correctness/PMU A/B. No Pi compilation, system/service mutation, -other quant format, BF16 kernel, model-forward or remote publication ownership. - **Interactive CLI chat/complete spike (`SERVE-CLI-CHAT`, 2026-08-01, `CLAIM-SERVE-CLI-CHAT-SPIKE`).** Codex (GPT-5), isolated worktree `/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`, @@ -1390,7 +1377,6 @@ table, tests, CMake. Details in the state-log entry of the same date. | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| -| `CLAIM-KERNEL-CPU-A76-Q8-DOT` | `KERNEL-CPU-A76-Q8-DOT` | Codex (GPT-5) | isolated worktree `/tmp/vllm-cpp-row-BACKEND-CPU`; local QEMU/buildx builder `pf-arm`; execution-only `rich@rpi5fan.lan:~/vllm-cpp-assembly` | `row/KERNEL-CPU-A76-Q8-DOT`, stacked on `16c848326` | New A76 Q8 C++/assembly TUs, narrow Q8 dispatch/header/CMake/test/harness edits, spike and checkpoint records. No Pi compilation, other quant/BF16/model-forward work or remote publication | `ACTIVE` | 2026-08-06, W0 spike; implementation and physical-Pi A/B next | | `CLAIM-KIMI-LINEAR-W0` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — dedicated W0 spike) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a771cc029e6843be0`; CPU-only, records-only — NO build, NO GPU, NO download (two GPU jobs queued ahead; the W0 GPU golden capture is a SEPARATE later step) | branch `worktree-agent-a771cc029e6843be0`, base `origin/main` HEAD `10dd23ee` | The FULL dedicated W0 spike for `KimiLinearForCausalLM` (Kimi-Linear-48B-A3B) per the spike-first protocol, so W1 implementation can start immediately. Owns ONLY: NEW `.agents/specs/kimi-linear.md`, the `MODEL-TEXT-kimi-linear-*` matrix row (Spike link → the dedicated spec + note; STAYS `SPIKE` because it is actively claimed — protocol forbids a claimed row from being `READY`; checklist mark stays `📋`), this claim row + the narrative block below, the roadmap breadth note, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, the `.agents/NOW.md` live-claim row + stamp, and one `.agents/state.md` entry. **NON-COLLISION:** records-only — touches NO model/kernel/registry/loader source, NO CMake, NO test. Co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` (MLA half) + `CLAIM-KDA-KERNEL` (KDA host refs), exactly as the K3 row is co-owned by `CLAIM-KIMI-K3-SCOPE`/`CLAIM-KIMI-K3-W2-W5`. | `DONE` | 2026-08-05 — **W0 DEDICATED SPIKE LANDED (records-only, NOT pushed).** Authoritative `config.json` fetched (27 layers = 20 KDA + 7 NoPE-MLA; 256e/top-8/1-shared sigmoid `noaux_tc` `routed_scaling=2.446` `first_k_dense_replace=1`; `num_nextn_predict_layers=0` ⇒ no MTP); reuse-vs-new map with our `file:line`; HW-fit (FITS one GB10, 91.5 GiB / 0.77× pool — real e2e SACRED gate reachable); W0-W7 breakdown + GPU golden-capture recipe. NET-NEW = KDA device kernel (host refs landed, `CLAIM-KDA-KERNEL`), NoPE-MLA branch, hybrid schedule/het-KV, loader name-map. Record checkers green (`check-model-checklist`, `check-agent-record`, `check-doc-checkpoint`, `check-now-current`). | | `CLAIM-KIMI-LINEAR-W1` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W1 registry/config/loader scaffolding; forward REFUSES-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a11ae231a71e8694b`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release); NO GPU, NO download — additive TU + one REGISTER line, forward is a `VT_CHECK(false)` stub | branch `worktree-agent-a11ae231a71e8694b`, base `origin/main` HEAD `053116df` | Kimi-Linear-48B-A3B W1 — registry + config + loader + KV-spec scaffolding so the W3-W6 forward can start. Owns ONLY: NEW `include/vllm/model_executor/models/kimi_linear.h`, NEW `src/vllm/model_executor/models/{kimi_linear_registry,kimi_linear_weights,kimi_linear}.cpp`, NEW `tests/vllm/models/test_kimi_linear_scaffold.cpp`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `KimiLinearForCausalLM` sorted-set + error-message + model-property inserts in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the forward refuses, rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` refuse stub so NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W1 CPU SCAFFOLDING LANDED (foreground, NOT pushed).** Additive registry (`kimi_linear`/`KimiLinearForCausalLM`, `is_hybrid`, `supports_multimodal=false`) + `ParseKimiLinearParams` (20 KDA + 7 NoPE-MLA schedule, MLA 512/128/64/128 q_lora-null, 256e/top-8/1-shared sigmoid `noaux_tc` 2.446, `first_k_dense_replace=1`; asserts `mla_use_nope`/`q_lora==null`) + `EnumerateKimiLinearTensors` name-map VERIFIED vs the real HF safetensors index (MoE = `block_sparse_moe.*`, the correction of the DERIVED K3 `mlp.*`) + loader THROWS BY NAME on missing/mis-shaped tensors + het KV spec (MLA latent-576 + KDA mamba `12288×3`/`32×128×128`) + REFUSE-by-name forward. `test_kimi_linear_scaffold` **9/9·83** + `test_model_registry` **24/24** + `test_kimi_k3_scaffold` 6/6 unchanged; clean CPU Release build; `check-fusion-consistency`/`check-runner-routing-consistency` green (refuse stub skipped); record checkers rc=0. RESIDUAL (W3-W6): KDA device kernel + NoPE-MLA route + sigmoid-noaux MoE + het-KV born-on-runner forward + the e2e SACRED gate (spec §8 recipe). | | `CLAIM-KIMI-LINEAR-W2` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W2-W6 CPU REFERENCE forward; the DEVICE runner forward stays refuse-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-aec13af094cd4692f`; CPU-only `build` (`-DVLLM_CPP_CUDA=OFF` RelWithDebInfo); NO GPU, NO download — additive forward TU + a host-materialization pass in the existing loader + additive test | branch `laguna-s21-w7-speed-profile` (worktree), base `origin/main` HEAD `0988db48` | Kimi-Linear-48B-A3B W2-W6 — the CPU REFERENCE forward, per-op gated, replacing the refuse-by-name host `KimiLinearModel::Forward`. Owns ONLY: NEW `src/vllm/model_executor/models/kimi_linear_forward.cpp`, NEW `tests/vllm/models/test_kimi_linear_forward.cpp`, its two CMake lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), ADDITIVE host-weight structs + per-op decls in `include/vllm/model_executor/models/kimi_linear.h`, the host-materialization pass in `src/vllm/model_executor/models/kimi_linear_weights.cpp`, the device-refuse doc/rename in `src/vllm/model_executor/models/kimi_linear.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the DEVICE forward refuses; rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + additive header structs; the host `Forward` becomes real but `ForwardDevice` (the DEFAULT gather_logits runner path) STAYS `VT_CHECK(false)` so the born-on-runner CI guard skips it (NO allowlist) and NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0/W1`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W2-W6 CPU REFERENCE FORWARD LANDED (foreground, NOT pushed).** Real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives: KDA layer (`vllm::kimi_kda` refs + the gated-delta recurrence `fused_recurrent.py:122-149`), NoPE-MLA (materialized-MHA reference, scaling `qk_head**-0.5`, no RoPE), sigmoid `noaux_tc` MoE (bias-select/unbiased-weight, renormalize, `routed_scaling=2.446`) + shared expert, dense layer-0 SwiGLU; loader now MATERIALIZES the host float weights (bf16/f32→f32). `test_kimi_linear_forward` **6/6·246** (KDA==kimi_kda composition; NoPE-MLA==materialized-MHA ref; router+block==hand-computed; loader materializes; whole 2-layer forward finite/coherent; greedy-decodes 5 tokens, context advances) + `test_kimi_linear_scaffold` 9/9·83 + `test_kimi_kda` 14/14 UNCHANGED; clean CPU build; `check-fusion-consistency`/`check-runner-routing-consistency` green (DEVICE forward refuse-skipped, no allowlist). RESIDUAL: the DEVICE born-on-runner forward + the W0/W7 e2e SACRED token golden on GB10 (spec §8 recipe) + speed. | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index 24340e2a6..d47ff89a7 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -274,7 +274,7 @@ evidence. |---|---|---|---|---| | `BACKEND-CUDA-SM121` | GB10/sm121a | `PARTIAL` | gate workload built, traced, token/perf gated; full component-family coverage is open | [backend row](backend-matrix.md#cuda-target-rows) | | `BACKEND-CUDA-OTHER` | vLLM sm70/75/80/86/87/89/90/100/101/103/110/120 targets | `ACTIVE` | 9 CUDA arches build-supported (sm80/86/87/89/90a/100a/103a/110/120a, single-arch portable-kernels-only, `-Werror` clean, SASS emitted); sm70/75/101 not build-supported; no non-121a target is runtime-validated | [backend matrix](backend-matrix.md), [CUDA inventory](specs/cuda-architecture-inventory.md) | -| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 R2-R3 is green: QEMU-built artifact, exact 16/16 x86-token and fixture parity, PMU baseline and a zero-loss model trace. Portable Q8 dot is 20.10% of user cycles; SDOT C++/assembly optimization and the Pi llama.cpp floor remain open | [backend matrix](backend-matrix.md), [RPi5 spike](specs/rpi5-cortex-a76-cpu-optimization.md), [harness](../examples/cpu_kernel_bench/main.cpp) | +| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 R4-R5 is green: QEMU-built artifact, exact operation/model output, and an AAPCS64 Q8 leaf that beats compiler SDOT 3.66-5.08% on M1/T1 and M128 while improving full-model E2E 0.13%. M1/T4 is −2.43%; Pi memory/concurrency and same-file llama.cpp floor remain open | [backend matrix](backend-matrix.md), [A76 Q8 dot](specs/cpu-a76-q8-dot.md), [binding evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md) | | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) | | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) | | `BACKEND-VULKAN` | Vulkan | `INVENTORIED` | runtime absent | [backend matrix](backend-matrix.md) | diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index e768a70af..0b566d661 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -112,7 +112,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner | |---|---|---|---|---|---|---|---| -| `KERNEL-CPU-A76-Q8-DOT` | Cortex-A76 Q8_0 x Q8_0 SDOT, exact-order C++ and scheduled AAPCS64 variants for the Pi decode/prefill keep-quant GEMM | llama.cpp `ggml/src/ggml-cpu/quants.c:400` portable dot and `arch/arm/quants.c:1076-1160` Arm DotProd @ project pin `237ad9b96`; Qwen semantics remain vLLM `555967922` | Baseline [portable dot](../src/vt/cpu/cpu_quant_dot.cpp#L88) and [quant GEMM caller](../src/vt/cpu/cpu_quant_gemm.cpp#L72); optimized provider not implemented at spike | Existing [Q8 upstream-test port and quant gates](../tests/vt/test_ops_quant_dot.cpp#L559); R3 zero-loss Pi model trace reaches portable Q8 dot at 20.10% of user cycles; assembly evidence pending | [A76 Q8 dot spike](specs/cpu-a76-q8-dot.md) | `SPIKE` | `CLAIM-KERNEL-CPU-A76-Q8-DOT` | +| `KERNEL-CPU-A76-Q8-DOT` | Cortex-A76 Q8_0 x Q8_0 SDOT, exact-order C++ and scheduled AAPCS64 variants for the Pi decode/prefill keep-quant GEMM | llama.cpp `ggml/src/ggml-cpu/quants.c:400` portable dot and `arch/arm/quants.c:1076-1160` Arm DotProd @ project pin `237ad9b96`; Qwen semantics remain vLLM `555967922` | [ACLE SDOT + HWCAP/MIDR selector](../src/vt/cpu/cpu_quant_dot_sdot.cpp#L85), [AAPCS64 two-block leaf](../src/vt/cpu/cpu_quant_dot_a76.S#L10), [Q8 dispatch](../src/vt/cpu/cpu_quant_dot.cpp#L675), [same-binary variants](../examples/cpu_kernel_bench/main.cpp#L858) | [direct exact variant/edge tests](../tests/vt/test_ops_quant_dot.cpp#L593); QEMU 20/20, 150258 assertions; physical Pi exact checksums/tokens; assembly vs compiler SDOT +3.66% M1/T1, +5.08% M128/T1, +3.69% M128/T4, but −2.43% M1/T4; [binding evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md#L40) | [A76 Q8 dot spec](specs/cpu-a76-q8-dot.md) | `GATING` | - | | `KERNEL-ACCEL-PROVIDER-SELECT` | **WHICH implementation of an op runs, when more than one exists on a device** — the selection layer above every kernel family in this matrix. Distinct from `KERNEL-CUDA-DISPATCH-AOT`, which is about which ARCH a CUDA kernel is compiled/selected for; this is about which PROVIDER (ours, a vendor library, MLX, llama.cpp) serves the op at all | no single upstream file: this is the shape vLLM's runtime chain uses everywhere — flashinfer tactic registries, cuBLASLt/CUTLASS per-call heuristics, and torch's backend selection — rather than compile-time pinning | `vt::OpProvider` [op_provider.h](../include/vt/op_provider.h) + [op_provider.cpp](../src/vt/op_provider.cpp); the flat `[OpId][DeviceType]` `void*` table it replaces is gone from [ops.cpp](../src/vt/ops.cpp) with the ~70 op wrappers untouched. Providers registered today: `vt-native` (every backend kernel in the tree, priority 0, unconditional — behaviour preserved exactly), `mlx` (priority 100, Metal `kMatmul`/`kMatmulBT`, build-gated `VLLM_CPP_MLX`, [metal_mlx_provider.mm](../src/vt/metal/metal_mlx_provider.mm)), and — NEW 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`, work row `S5`) — **`vt-cpu-ref` (priority −1000, the portable reference tier)**: the CPU kernel installed LAZILY as a negative-priority fallback on a UNIFIED-MEMORY device's first `GetOp` miss, mirroring `custom_op.py:138 forward_native`, so a partial backend runs an op it lacks natively instead of throwing. Native always wins (priority); gated on `Backend::UnifiedMemory()` (a discrete GPU never gets it — a CPU kernel on true device memory is corruption); observable via `GetReferenceTierHits()` + a one-time loud stderr line | [test_op_provider.cpp](../tests/vt/test_op_provider.cpp) 11 cases / 47 assertions — deterministic selection under REVERSED registration order, name tie-break, duplicate rejection, capability predicate, caps re-resolution, decline-and-fall-back, stats, runtime disable; [test_metal_backend.cpp](../tests/vt/test_metal_backend.cpp) 9 cases / 108 assertions on the M4 with MLX ON, including MLX-vs-MSL-vs-CPU NMSE per op at real shapes and an end-to-end DECLINE; **[test_reference_tier.cpp](../tests/vt/test_reference_tier.cpp) (S5): discrete-device refusal + unified-device zero-native-kernel fallback correctness + native-wins + observability, hardware-free via a fake backend on `kXPU`.** Linux CPU 156/156; dgx regression set ALL UNCHANGED — anchor `tests/vt/test_op_provider.cpp:64` | [Metal/MLX reuse study §6](specs/metal-mlx-reuse-study.md); [accelerator-seam-audit §10](specs/accelerator-seam-audit.md); [drop-in kernel ABI](specs/dropin-kernel-abi.md) (the complementary ARGUMENT half) | `ACTIVE` — mechanism landed and gated with THREE provider kinds (`vt-native`, `mlx`, `vt-cpu-ref`); the CUDA/CPU/Vulkan vendor provider rows it was designed for are not yet populated (so the row is deliberately left open) | `CLAIM-BACKEND-ACCEL-PROVIDER-1` | | `KERNEL-CUDA-DISPATCH-AOT` | CUDA runtime, streams, graphs, per-arch dispatch, and generated/AOT artifact selection | platform selection `vllm/platforms/cuda.py:205-493`; gencode handling `CMakeLists.txt:201-220`; JIT packages `cmake/external_projects/triton_kernels.cmake:1-28` | [cuda_backend.cu:20](../src/vt/cuda/cuda_backend.cu#L20), [CMakeLists.txt:37](../CMakeLists.txt#L37), [TritonAOT.cmake:57](../cmake/TritonAOT.cmake#L57) | [CUDA backend tests](../tests/vt/test_cuda_backend.cpp#L31); the Triton GDN AOT tree is now vendored per-arch — `sm_121a` (RUNTIME-VERIFIED) plus `sm_80/86/89/90a/100a` (DERIVED+BUILD-VERIFIED, `CLAIM-TRITON-AOT-PER-ARCH`, 2026-07-28), each a real per-target cubin (`cuobjdump sm=80/86/89/90/100`); the CUDA fast-path arch fan-out (non-GDN) remains as evidenced in `backend-matrix.md` | [inventory](specs/kernel-family-inventory.md); [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `ANCHOR-BACKFILL` | - | | `KERNEL-FUSION-FRAMEWORK` | **Portable automatic op-fusion framework (roadmap_v1 ORDER-1 extensibility cornerstone)** — declare-once `constexpr FusedRecipe` catalog (backend-agnostic, above `vt::`) transcribing vLLM's finite fusion-pass set; realize-per-backend through the `vt::` op table (Tier-0 composite = CPU oracle inherited free; Tier-1 interpreter = one kernel/backend lights up every recipe). A new vLLM fusion PR = ONE declaration; a new GPU = one catalog realization; a new model = additive pattern bindings. NOT a perf lever (35B ceiling ~3.5%/step compute-bound); primary value = extensibility + mechanical upstream-sync + CPU/CUDA oracle-drift elimination | pass manager `vllm/compilation/passes/pass_manager.py:86,138-200`; the finite fusion-pass set `passes/fusion/{rms_quant_fusion.py:618,act_quant_fusion.py:283,qk_norm_rope_fusion.py:188,attn_quant_fusion.py:362}`; declare-once analog `vllm/model_executor/custom_op.py:103` (`forward_native`/`forward_cuda`/`enabled()` :138-207); combo_kernels OUT-OF-SCOPE `vllm/config/compilation.py:967` | Phase-0 skeleton LANDED: [fused_recipe.h:72](../include/vt/fused_recipe.h#L72), [recipes.h:35](../include/vt/recipes.h#L35), [kFusedChain dispatch](../src/vt/ops.cpp#L639), Tier-0/1 CUDA [cuda_ops.cu:1062](../src/vt/cuda/cuda_ops.cu#L1062) + CPU [cpu_ops.cpp:1427](../src/vt/cpu/cpu_ops.cpp#L1427); catalog-to-be = the hand-fused ops (`kRmsNormQuantFp8`/`kSiluMulFp4Quant`/`kAttnQkNormRopeGate`/`kSigmoidGateFp4Quant`/`kRmsNormGatedQuantFp8`). **2026-07-19 (`CLAIM-FUSION-FRAMEWORK-W0`): W0 ADOPTED at ONE production site** — `RunLayerPaged` post_attention_layernorm (35B MoE paged, `test_qwen36_paged_engine` hot path) now routes the plain add+residual+gemma-RMSNorm through [`vt::FusedChain(kFusedAddRmsNorm)`](../src/vllm/model_executor/models/qwen3_5.cpp#L4784) behind [`FusedChainAdoptEnabled()` (`VT_FUSED_CHAIN_ADOPT`, default ON / `=0` restores the exact prior `vt::RmsNorm(residual)` hand-call)](../src/vllm/model_executor/models/qwen3_5.cpp#L1394). BEHAVIOR-PRESERVING by construction (Tier-0 composite dispatches to the same `vt::RmsNorm(residual)` primitive; perf-neutral, NOT a speedup) — proves the declare-once/realize-per-backend seam end-to-end in production. The other 3 identical plain sites (`RunLayer`/`RunDenseLayer`/`RunDenseLayerPaged` post-attn + the input-layernorm plain path) are left on the hand-call; W1's generalized POD (multi-input operands + quant/silu/rope opcodes) is required only for the quant-fused sites (`kSiluMulFp4Quant`/`kRmsNormQuantFp8`/…), NOT for these plain add+rmsnorm sites — the current 3-opcode/4-role POD expresses them byte-identically. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W1`): W1 GENERALIZED the POD (infrastructure only, perf-neutral, NO call site changed).** `FOp` grew from `{kAdd,kMul,kRmsNorm}` to add `kSilu`/`kSigmoid` (Tier-1 elementwise) + fused-primitive opcodes mapping 1:1 to standalone ops so the composite is byte-exact by construction — `kSiluMul`→`vt::MoeSiluMul`, `kSigmoidGate`→`vt::SigmoidGateBf16`, `kRmsNormGated`→`vt::RmsNormGated`, `kRope`→`vt::RopeFromCache`, `kQuantFp8`→`vt::QuantFp8Static`, `kQuantFp4`→`vt::ScaledFp4Quant`, `kAttnQkNormRopeGate`→`vt::AttnQkNormRopeGate`. The fixed `{kIn,kResidual,kWeight,kOut}` 4-role model became an INDEXED `FOperandSlot operands[8]` table (`FStep` names input indices + `out`/`out2`); tensors bind positionally via `FusedBinding`, scalars via `FusedParams`; intermediates are caller-bound scratch (byte-exact to the unfused sequence). Tier-0 composite is now ONE device-agnostic walker in `ops.cpp` (per-opcode dispatch to the standalone op, self-dispatching per device — eliminates CPU/CUDA oracle drift); the per-backend `kFusedChain` op is now ONLY the Tier-1 interpreter over the canonical `[x,weight,residual,out]` shape. All five W2 target chains DECLARED as `constexpr` recipes ([recipes.h](../include/vt/recipes.h), citing each vLLM pass). TIER (honest): composite realizes all five byte-exact; Tier-1 covers only the elementwise/rmsnorm subset (W0 `kFusedAddRmsNorm`) — the four quant/activation chains + attn preamble are COMPOSITE-ONLY. Backend-negotiation (§3b/§6): fp8 terminal `vt::QuantFp8Static` CUDA-only (fp8 chains' composite CUDA-only; CPU covers up to the bf16 norm); `kAttnQkNormRopeGate` a composite-only MACRO (per-head 3-D operands → dispatches the whole preamble to the single `vt::AttnQkNormRopeGate` op). ALL FIVE byte-exact expressible, none forced. Anchors: generalized POD [fused_recipe.h](../include/vt/fused_recipe.h), binding/params/general entry [ops.h](../include/vt/ops.h), device-agnostic composite [`FusedChainComposite`](../src/vt/ops.cpp), Tier-1 interpreters [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)+[cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W2`): W2 MIGRATED the bespoke hand-fusions to `vt::FusedChain(recipe)` — the framework now OWNS the fusion dispatch, perf-neutral by construction.** New realization-binding infra (the "fast realization" tier the spike §3c reserved): `FusedRecipe.fast_op` ([fused_recipe.h](../include/vt/fused_recipe.h)) carries the raw `OpId` of the EXISTING single-launch bespoke fused kernel that realizes the whole recipe (backend-AGNOSTIC — names WHICH abstract op, realized per-backend via the op table); a non-throwing `OpRegistered` probe + a per-recipe `DispatchFusedFast` adapter switch + convenience overloads in [ops.cpp](../src/vt/ops.cpp)/[ops.h](../include/vt/ops.h). `FusedChain(recipe,…)` dispatches recipe→fast_op WHEN the backend registers that OpId (the SAME kernel the model called directly pre-migration → byte-identical AND perf-neutral; no extra kernel, no per-forward getenv/alloc), else falls through to the byte-exact Tier-0 composite (graceful degradation). The composite is the exposed oracle ([`FusedChainComposite`](../src/vt/ops.cpp)) the fast realization is validated against. Bound recipes: `kRmsNormQuantFp8`→`OpId::kRmsNormQuantFp8`, `kRmsNormGatedQuantFp8`→`OpId::kRmsNormGatedQuantFp8`, `kSiluMulFp4Quant`→`OpId::kSiluMulFp4Quant`, `kSigmoidGateFp4Quant`→`OpId::kSigmoidGateFp4Quant` ([recipes.h](../include/vt/recipes.h)); `kAttnQkNormRopeGate` has NO fast_op (its composite MACRO already dispatches to the single `vt::AttnQkNormRopeGate` launch — perf-neutral trivially). SIX call sites migrated in [qwen3_5.cpp](../src/vllm/model_executor/models/qwen3_5.cpp) behind `VT_FUSED_CHAIN_ADOPT` (`=0` restores the exact prior hand-calls, same binary): `kSiluMulFp4Quant` (MoE down-proj), `kSigmoidGateFp4Quant` (full-attn o-proj), `kRmsNormGatedQuantFp8` ×2 (GDN out-proj), `kRmsNormQuantFp8` (input-layernorm), `kAttnQkNormRopeGate` ×2 (attn preamble). qwen3_5.cpp net +61 lines (−21/+82): each migrated fused-op call became one `FusedChain` call (call-for-call neutral), but the mandated same-binary rollback `else` branch + explanatory comment at each of the 6 sites grows the raw count — the true shrinkage lands when the rollback branches and bespoke `OpId`s retire; the dispatch surface is already framework-OWNED. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W3`): W3 MECHANICAL-UPSTREAM-SYNC PROOF** — ported a NEW, previously-unported vLLM fusion-pass variant, `SiluMulFp8StaticQuantPattern` (`vllm/compilation/passes/fusion/act_quant_fusion.py:81` → `_C.silu_and_mul_quant`; registered unconditionally at `act_quant_fusion.py:296`), as ONE `constexpr FusedRecipe kSiluMulQuantFp8` ([recipes.h](../include/vt/recipes.h)) + its byte-exact test. The static-per-tensor-FP8 sibling of `kSiluMulFp4Quant`; its Tier-0 composite is realized ENTIRELY from EXISTING standalone `vt::` ops (`kSiluMul`→`vt::MoeSiluMul`, then `kQuantFp8`→`vt::QuantFp8Static`) — NO new primitive, NO kernel, NO composite-walker case, NO dispatch edit, NO model-site edit. `fast_op = kNoFastOp` (no bespoke silu·mul→static-fp8 fused OpId exists in-tree; a fast kernel is a separate later perf step). **Additivity headline: adding a whole new fusion pattern touched exactly 2 shared files** — `include/vt/recipes.h` (+1 declaration) + `tests/vt/test_ops_fused_chain.cpp` (+1 byte-exact test); `git diff --stat main` = those 2 files only. This IS the framework's primary-value proof (spike §4/§10): a new upstream fusion PR = one declaration. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W4`): W4 — the EXECUTABLE BACKEND-ADDITIVITY PROOF, closing the W-series.** Makes the claim executable: "a new backend registers `kFusedChain` (Tier-0 composite) ONCE and inherits the ENTIRE catalog correct, ZERO per-recipe work." Proof approach (spec §10 W4, PREFERRED option b): treat the EXISTING CPU backend AS the 'second backend' relative to CUDA — NO mock `DeviceType` (which would edit the core enum + every switch, ironically non-additive). New test [`test_fused_chain_additivity.cpp`](../tests/vt/test_fused_chain_additivity.cpp) enumerates the WHOLE catalog (`kCatalog`, all 7 recipes) and in ONE generic loop asserts each runs correct on the CPU backend via the Tier-0 composite ([`vt::FusedChainComposite`](../src/vt/ops.cpp)), BYTE-EXACT vs the standalone-op-sequence golden, over the CPU-expressible scope: 4 CPU-FULL end-to-end (`kFusedAddRmsNorm`/`kSiluMulFp4Quant`/`kSigmoidGateFp4Quant`/`kAttnQkNormRopeGate`), 3 CPU-PREFIX (`kRmsNormQuantFp8`/`kRmsNormGatedQuantFp8`/`kSiluMulQuantFp8` — static-fp8 terminal `vt::QuantFp8Static` CUDA-only per §3b/§6; the prefix is byte-exact and the FULL composite is asserted to THROW on CPU, documenting the backend-negotiated tail, not silently skipping). ADDITIVITY EVIDENCE (catalog GROWS ⇒ backend does NOT): catalog [recipes.h](../include/vt/recipes.h) 1→6→7 recipes (W0→W1→W3); the composite walker [`FusedChainCompositeImpl`](../src/vt/ops.cpp) is ONE function with a per-OPCODE switch (12 `FOp::` cases, NOT per-recipe); the CPU/CUDA `kFusedChain` registration is ONE line each; `src/vt/cpu/cpu_ops.cpp` does NOT `#include recipes.h` (the backend never sees the catalog); W3's whole new recipe `kSiluMulQuantFp8` appears in ZERO backend TUs + ZERO in the `ops.cpp` dispatch — it runs purely through the shared walker (that is WHY W3 touched 2 files and no `src/vt/`). **KERNEL-FUSION-FRAMEWORK ORDER-1 milestone DONE** (W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity). HONEST DEFERRED / future (named, non-blocking the ORDER-1 milestone): (a) the Tier-1 PERF interpreter covers only the elementwise/rmsnorm subset — the quant/activation chains realize composite-only (correct/byte-exact, but the single-pass fused perf kernel for the quant chains is future); (b) a REAL Metal/Vulkan realization of the catalog needs the M4 dev-box (HW-blocked); (c) per-recipe fast single-launch kernels for composite-only recipes are separate later perf steps. Perf is NOT the framework's value (§11: 35B ceiling ~3.5%/step, compute-bound). **2026-07-25 (`CLAIM-FUSION-CONSISTENCY-AUDIT`): CONSISTENCY AUDIT + ENFORCEMENT** — read-only static audit of catalog USAGE across all 12 model forwards ([fusion-consistency-audit.md](specs/fusion-consistency-audit.md)). Verdict MOSTLY consistent: qwen3_5 family fully migrated; qwen3/qwen3_moe/deepseek_v2 adopted `kFusedAddRmsNorm{,Std}`; **DRIFT — gemma/gemma2/gemma3/glm4/phi3 hand-call the residual `vt::RmsNorm(...,&res)` add+RMSNorm chain and never route through the catalog** though the recipe already exists (one-line adoption each = follow-on `FUSION-DENSE-MIGRATE`). olmo2/granite/opt deliberately-not-fused (post-norm/LayerNorm). GDN glue + MoE-combine remain bespoke (spike §10 W2 remainder). NEW enforcement CI check [`scripts/check-fusion-consistency.py`](../scripts/check-fusion-consistency.py) (+ mutation test, wired into the `agent-record` CI job) flags any model that hand-fuses add+RMSNorm without the catalog unless allowlisted — the 5 drift models on [`fusion-consistency-allowlist.txt`](../scripts/fusion-consistency-allowlist.txt) keep the gate GREEN while blocking new silent bypasses; removing a stem is the gate closing. | [test_ops_fused_chain.cpp](../tests/vt/test_ops_fused_chain.cpp) (byte-exact composite==interp==golden; W1 extends it: each new recipe's Tier-0 composite == its standalone-op-sequence golden byte-exact, fp4+attn CPU+CUDA, fp8 CUDA-only. **W2 adds the fast==composite==golden triple arm per recipe — CPU 228 assertions, CUDA-on-dgx 420, memcheck 0**). **W3 adds `RunSiluMulQuantFp8Cuda` (kSiluMulQuantFp8: FusedChain==composite==MoeSiluMul+QuantFp8Static golden, byte-exact, CUDA — the newly-ported pass's test), and the no-token-regression gate `test_qwen27_paged_engine` 235/235 + `test_qwen36_paged_engine` 315/315 unchanged (the new recipe is DECLARED, not wired into any model — the engine is untouched)**; **W4 adds `test_fused_chain_additivity.cpp`** (the whole-catalog additivity loop: every recipe byte-exact on the CPU 'second backend' via the Tier-0 composite, CPU-full end-to-end / fp8-prefix + negotiated-tail-throw; count guard `==7`) — **1 case / 17 assertions CPU green, sibling `test_ops_fused_chain` 228/228 unchanged, engine byte-identical so 27B 235/235 + 35B 315/315 structurally unchanged**; to port: `tests/compile/passes/test_fusion.py:249-287` oracle discipline (byte-exact, stricter than upstream `assert_close`) — anchor `tests/vt/test_ops_fused_chain.cpp:124` | [portable-fusion-framework.md](specs/portable-fusion-framework.md) | `ANCHOR-BACKFILL` (W-series ORDER-1 PROOF MILESTONE DONE: W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity, all merged+gated; row stays SPIKE-open because the framework retains named future work — the Tier-1 quant-chain perf interpreter (composite-only today) + a real Metal/Vulkan catalog realization (M4 HW-blocked) + per-recipe fast kernels, §10 Wn) | `CLAIM-FUSION-FRAMEWORK-W4` | diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 44ab5cff4..f27069879 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -915,3 +915,4 @@ Columns: | 2026-08-06 (`row/KERNEL-MARLIN-DENSE-PORT`; `CLAIM-KERNEL-MARLIN-DENSE-PORT`; kernel `KERNEL-GEMM-MARLIN-W4A16`; gated default-OFF, lifecycle unchanged) | Vendors vLLM's OWN dense marlin W4A16 GEMM as a new `vt::MarlinDenseGemm` op (`VT_MARLIN_DENSE`, default OFF) and routes the E=1 dense NVFP4/MXFP4 projections (`dense_nvfp4_gemm.h` `MatmulNvfp4MarlinD`/`MatmulMxfp4W4A16D`/`GateUpFusedMarlinD`) through it. The dense kernel is direct-A + tile-per-CTA with vLLM's OWN dense fp32-C_tmp reduce, so at M<=8 it runs the sms-wide (48-CTA) grid WITHOUT the one-bf16-ULP shift the `VT_MARLIN_E1_PAR1` MoE-route par-regroup costs (#54: that ULP flips a strict 32B-NVFP4A16 token). Reuses the EXISTING marlin resident + workspace (same `marlin_permute` repack for dense and MoE — confirmed, no shim); rank-2 operand views, no moe_align gather. | 1:1 lift of vLLM @ `555967922` `csrc/libtorch_stable/quantization/marlin/`: `marlin.cu:326-541` (`marlin::marlin_mm` + config helpers) → `marlin_mm_dense.cu`; the torch::stable `marlin_gemm` wrapper (`:545-894`) → torch-free `cuda_marlin_dense.cu` launcher (mirrors `cuda_moe_marlin.cu`, dense c_tmp sizing `:713-716`); `kernel.h`/`marlin_template.h:1-2081` verbatim (the DENSE kernel — DISTINCT from the moe one, but SAME 12-param `Marlin<>` template so the generated `kernel_selector.h`+`sm80_*.cu` instantiation set is shared, namespace `marlin` from the local kernel.h). Shared `marlin.cuh`/`marlin_dtypes.cuh`/`dequant.h`/`marlin_mma.h` diff-verified byte-identical. Forced-Marlin a16 selection `kernels/linear/__init__.py:879-881`. | CPU `-fsyntax-only` CLEAN (`ops.cpp` + the `VT_MARLIN_NVFP4` routing header). GPU compile: all 3 new dense `.cu` compile CLEAN on dgx GB10 sm_121a under exact production flags (`-Werror=all-warnings`, `-static-global-template-stub=false`, `--generate-code=…sm_121a`). RED-first unit battery WRITTEN (`test_ops_moe_grouped.cpp`: NVFP4+MXFP4, M=1..8 × 3 shapes, dense-vs-CPU-ref AND dense-vs-grouped-route, row-shifted stride RED-injection). `benchmark_binding=false`; GPU EXEC gates (unit run + strict token battery dense-ON vs oracle incl. 32B-NVFP4A16:344 + launch-counter + nsys 48-CTA + binding c1..c8 x3) are the scoped dgx follow-up; default stays OFF until the strict battery proves oracle byte-match and the binding beats the MoE route (state `KERNEL-MARLIN-DENSE-PORT`). | | 2026-08-06 (`row/H3-FP4-SPEED`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; lifecycle unchanged) | **MiniMax-H3 W-FP4a — fp4-RESIDENT NVFP4 routing for the device DiT forward (NO new quant code).** Until now both NVFP4 loaders dequantized the packed FP4 projections to bf16 and the device forward ran `vt::MatmulBT`, so the sm_121a FP4 tensor-core route never ran for H3. Adds `Nvfp4Weight` carriers to `MiniMaxH3DitBlockWeights`/`MiniMaxH3DitWeights`, a fp4-resident streamer `StreamMiniMaxH3Nvfp4ToDeviceFp4` (keeps the compressed-tensors triple host-resident; the shared dispatcher uploads+repacks lazily then frees the fp4 originals, so peak device memory is ~1/4 of the bf16 arm), and a `LinearDev` dispatch that routes a non-Empty fp4 projection through `dense_nvfp4::MatmulNvfp4W4A16D`. | The routing is vLLM's OWN forced-Marlin-for-a16 selection: the checkpoint is weight-only NVFP4 (no `input_activations`, `IsTrueW4A4()==false`), so `kernels/linear/__init__.py:879-881` forces the Marlin W4A16 kernel, mirrored by `include/vllm/model_executor/models/dense_nvfp4_gemm.h:12-22,505-549` (`MatmulNvfp4W4A16D` -> single-expert `vt::MoeGroupedGemmNvfp4Marlin`, the SAME kernel Laguna routed-experts + dense Qwen3-32B NVFP4 use). Not cutlass-fp4/W4A4 (needs fp4 activations, private to `qwen3_5.cpp`). fc1 is already merged `[gate;up]` -> one W4A16 GEMM + `vt::SiluAndMul`. | **CPU-GATED (wiring), verified**: `test_minimax_h3` 62/62 cases / 30039 assertions, 0 failed. The synthetic-NVFP4 case streams the fp4 twin, asserts the loader kept the projections PACKED (fp4 slot set / bf16 slot Empty, and the inverse for the bf16 loader), runs fp4 + bf16 device forwards on the SAME file, asserts the W4A16 dispatcher executed ALL 11 quantized GEMMs (the `Nvfp4W4A16Stats` this-path-ran counter), and bounds fp4-vs-bf16 <= 2e-3. On CPU the dispatcher has no Marlin op so it falls to the bf16 arm's own dequant+matmul (hence a WIRING gate here); the Marlin kernel numerics are CUDA-gated independently by `test_ops_nvfp4_matmul` / `test_linear_method` (2e-3 f32-out / 8e-3 bf16-out vs a bf16 reference). `benchmark_binding=false`. PENDING: GB10 CUDA build + the fp4-vs-bf16 numeric delta and steady per-step timing (disk/build window); real-checkpoint t2va e2e DISK-BLOCKED (~41 GB working set). Comparability: vLLM-Omni serves NO quantized H3 (BF16-only in practice; source-audited `a4ea67a2`, spec §8.3) -> HW/loader-forced-indirect. | | 2026-08-06 (`row/BACKEND-CPU`; `BACKEND-CPU` R1; PR #65; lifecycle remains `PARTIAL`) | Adds `vllm-cpu-kernel-bench`, a developer-only vt-op benchmark substrate: deterministic quant-GEMM fixtures, calibrated batched timing, cache-pressure profiles, affinity, JSON, checksums, system metadata, and grouped generic/Cortex-A76 `perf_event_open` counters with explicit multiplex/unsupported status. Production dispatch and numerics are unchanged. | No vLLM behavior counterpart; vLLM remains the x86 semantic oracle and llama.cpp `237ad9b96` remains the Pi performance floor. The quant fixture invokes the existing `vt::MatmulBTQuant` contract unchanged. | **CPU-GATED, `benchmark_binding=false`.** GCC 15.2 `-Wall -Wextra -Werror` build; clang-format clean; `test_cpu_kernel_bench_cli` deterministic JSON schema/checksum + invalid-input + structured-counter cases; direct 1/4-thread x86 runs and real generic PMU counts. X86 timings are non-binding tool validation. Pi PMU execution, model correctness, throughput and memory all remain `PENDING`. | +| 2026-08-06 (`KERNEL-CPU-A76-Q8-DOT` R4-R5; `CLAIM-KERNEL-CPU-A76-Q8-DOT`; physical RPi5 Cortex-A76; closing commit: this checkpoint) | Adds an exact-order ACLE SDOT control and an original AAPCS64 two-block Q8_0×Q8_0 leaf behind Linux DotProd/MIDR dispatch. `auto` selects assembly only on Cortex-A76+DotProd; x86, non-DotProd and other Arm CPUs retain portable dispatch. Explicit `portable`/`sdot`/`a76-asm` same-binary controls remain. ARM64 builds/tests locally through buildx/QEMU; Pi is execution-only. | The integer structure is informed by llama.cpp `237ad9b96` `ggml/src/ggml-cpu/arch/arm/quants.c:1076-1160`, while this port deliberately retains the local portable function's per-block f32 reduction order. vLLM `555967922` supplies Qwen3.5 semantics, not a corresponding CPU microkernel. Local anchors: `src/vt/cpu/cpu_quant_dot_{sdot.cpp,a76.S}`, Q8 dispatch in `cpu_quant_dot.cpp`, direct tests in `tests/vt/test_ops_quant_dot.cpp`, and [immutable evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). | **PASS for the compiler-gap/component gate; row `GATING`, `benchmark_binding=true`.** Final binaries `9eb57cf...`/`a94dad30...`; QEMU focused suite 20/20, 150258 assertions; physical-Pi checksums exact. Assembly vs compiler SDOT wall/cycles/instructions: M1/T1 +3.66%/+3.17%/+10.10%, M128/T1 +5.08%/+4.61%/+10.24%, M128/T4 +3.69%/+3.69%/+9.74%. M1/T4 is an explicit −2.43% wall/−4.32% cycles residual despite 8.77% fewer instructions. All 64 Qwen tokens equal the x86 golden in all nine runs; median assembly vs SDOT TTFT −1.55%, TPOT −0.05% neutral, E2E −0.13%. Disassembly proves GCC's framed dependent one-block loop versus the stack-free independent two-block schedule. Same-file Pi llama.cpp, peak memory and concurrency remain `PENDING`; no competitor-floor binding is claimed. | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 0ec7c7ef0..9ffb1520d 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -68,7 +68,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 7 | `ROAD-V1-C7` | Sampling/API controls and logprobs payloads | [engine matrix](engine-matrix.md), [coverage view §6](feature-matrix.md#6-sampling--generation-controls), [C7 spec](specs/sampling-controls-c7.md) | **W1-W4 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7`, NOT pushed): the full sampling-control surface is WIRED end-to-end + gated exactly on the CPU reference backend.** `SAMPLE-CORE` + `SAMPLE-LOGIT-FILTERS` -> `ACTIVE`: SamplingParams undefers + validates logit_bias/allowed_token_ids/bad_words + all_stop_token_ids; OpenAI protocol parses the three fields + clamps logit_bias to [-100,100]; InputBatch tracks + `build_sampling_metadata` populates min_p/min_tokens/logit_bias/allowed_token_ids_mask/bad_words/max_num_logprobs with condense/swap maintenance; InputProcessor tokenizes bad_words + seeds the eos stop-set. RED-first (disabling a wiring line fails the gate); inertness proven (default/greedy build byte-identical SamplingMetadata, SACRED greedy unaffected); CPU `-Werror` 0-warn. The transforms are device-neutral pure functions gated exactly (not near-tie). **W5 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7-LOGPROBS`, NOT pushed): the logprobs PAYLOAD is wired end-to-end + gated on the CPU reference engine.** `SAMPLE-LOGPROBS` -> `DONE`: ported `vllm/logprobs.py` + `vllm/v1/engine/logprobs.py` (LogprobsProcessor); threaded ModelRunnerOutput.logprobs -> scheduler slice_request -> EngineCoreOutput.new_logprobs -> OutputProcessor -> CompletionOutput.logprobs -> OpenAI `CompletionLogProbs`/`ChatCompletionLogProbs` serialization (grounded 1:1 in vLLM 0.26 `555967922`). Gate: serialization vs a hand-computed vLLM oracle (completion N+1 vs chat N cutoff, bytes, text_offset, -9999 floor) RED-first, + e2e through the CPU Qwen3.6 engine (`logprobs=K` greedy sampled==argmax==rank1, chat `top_logprobs`, null-when-off inertness); CPU `-Werror` 0-warn. **C7-core (SAMPLE-CORE + SAMPLE-LOGIT-FILTERS + SAMPLE-LOGPROBS) is complete.** | `DONE` | **REMAINING (separate rows, NOT required for C7-core):** `SAMPLE-PROMPT-LOGPROBS` payload/serialization is done but its runner prompt-position-logits SOURCE is a runner/prefill addition (row `PARTIAL`); plus INVENTORIED `n>1` execution, `SAMPLE-PHILOX` exact RNG, `SAMPLE-LOGPROB-TOKEN-IDS`/logprobs_mode, beam search. **Custom logits processors LANDED 2026-07-28 (`SAMPLE-CUSTOM-PROCESSORS` -> `ACTIVE`, `CLAIM-C7-CUSTOM-LOGITS`): per-request C-ABI callback (`vllm_logits_processor`, ABI v8) at the sampler's non-argmax-invariant stage; default byte-identical; also closes SGLang `SGLANG-SAMPLING-CUSTOM`.** **Parallel sampling `n>1` LANDED 2026-07-28 (`SAMPLE-N` -> `ACTIVE`, `CLAIM-C7-N-SAMPLING`): the OpenAI `n` parameter now EXECUTES — an `n>1` request fans out into n prompt-sharing child sequences (`ParentRequest`/`FanOutParallelSampling`, mirroring vLLM `parallel_sampling.py`) aggregated into one `RequestOutput` / n indexed `choices`; `n==1` byte-identical; CPU-gated RED-first (1→n) with a `top_k=1` determinism gate. Residuals: `best_of`, async-streaming per-child collation, C-ABI `n`.** **Beam search LANDED 2026-07-28 (`SAMPLE-BEAM` -> `ACTIVE`, `CLAIM-C7-BEAM`): beam search as an OUTER engine loop (mirroring vLLM `entrypoints/generate/beam_search/{utils,offline}.py`) — per step one decode per beam (`logprobs=2*beam_width`), expand + score by `get_beam_search_score` (cum_logprob / `seq_len**length_penalty`), keep top-`beam_width`, retire EOS, return `beam_width` outputs (reuses the `SAMPLE-N` multi-output seam). Deterministic ⇒ token-EXACT: model-free gate vs a hand-computed beam tree (RED-first 1→2 beams) + e2e over the CPU engine (`bw=1`==greedy). Residuals: OpenAI `use_beam_search`/`best_of` wiring, C-ABI beam params, grammar-constrained beams.** **`best_of` + `use_beam_search` OpenAI-ENDPOINT surface LANDED 2026-07-28 (`SAMPLE-BEST-OF` -> `ACTIVE` + `SAMPLE-BEAM` endpoint, `CLAIM-C7-BESTOF-BEAM-API`): both request types gain `best_of`/`use_beam_search`/`length_penalty` + `to_beam_search_params`; `best_of>n` fans out via `SAMPLE-N` and returns the top-n by cumulative logprob (`SelectBestOf`); `use_beam_search` routes through the merged `BeamSearch` driver over the SYNC `LLMEngine` seam and returns `beam_width` beams as choices (endpoint output IDENTICAL to the direct driver call, gated completion + chat). Default (no best_of/beam) path byte-identical. HONEST FINDING: vLLM 0.26 exposes `use_beam_search` on the OpenAI server but has DROPPED `best_of` (only a vestigial `BatchChatCompletionRequest` field) — best_of follows classic OpenAI/V0 semantics gated on our own fan-out. Residuals: async-HTTP beam (driver is `LLMEngine&`-based, production server is AsyncLLM), streaming beam (rejected like upstream), C-ABI best_of/beam.** **Async/production beam LANDED 2026-07-28 (`SAMPLE-BEAM` async coverage, `CLAIM-C7-BEAM-ASYNC`): `BeamSearchAsync(AsyncLLM&, …)` mirrors vLLM `entrypoints/generate/beam_search/online.py:28-220`, drives the production AsyncLLM (the engine the HTTP server holds) per-beam single-token `generate` and calls the SAME merged `BeamSearchStep`/scoring (shared template driver; only the engine differs). `use_beam_search` on the server now RUNS instead of raising "requires the synchronous engine"; gated token-IDENTICAL to the sync `BeamSearch` (bw 1/2/3) + endpoint choices IDENTICAL over the AsyncLLM backend (completion + chat). Per-beam decodes driven SEQUENTIALLY (byte-identical to sync); concurrent per-beam stepping (online.py `asyncio.gather`) is a named residual. Remaining beam residuals: streaming beam, C-ABI beam, grammar-constrained beams.** | | 8 | `ROAD-V1-C8` | Tokenize/detokenize, unified streaming parsing and full metrics | [engine matrix](engine-matrix.md), [coverage views §§7,9](feature-matrix.md#7-structured-outputs--tool-calling), [metrics spec](specs/prometheus-metrics.md), [endpoints spec](specs/utility-endpoints.md) | **`SERVE-METRICS` + `SERVE-UTILITY-ENDPOINTS` LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8`, NOT pushed):** the oldest T0 debt — Prometheus `/metrics` (self-contained registry + text-0.0.4 exposition + always-on vLLM metric catalog, names/labels/buckets 1:1) — is CLOSED and gated by the vLLM scrape spec `EXPECTED_METRICS_V1` (RED-first substring gate, `test_prometheus_metrics` 4/4/81); plus `/tokenize`,`/detokenize`,`/ping`,`/server_info`,`/reset_prefix_cache` (additive/opt-in, schema-matched, `test_openai_api_server` 26/26/277). Inertness: opt-in, existing serving byte-identical. **`TOOLS-STREAMING-PARSER` core LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-PARSER`, NOT pushed):** the vLLM 0.26 unified streaming parser ENGINE (declarative `StreamingParserEngine`: token-ID scanner + prefix-buffering lexer + transition state machine + JSON-arg hold-back + drop-info) with qwen3/seed_oss/kimi_k2 configs + the unified registry, EXACT-gated event-for-event vs vLLM 0.26 (`test_streaming_parser_engine` 586/586, RED-first, goldens byte-reproduced from the pin). **Parser ASSEMBLY layer LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-ASSEMBLY`, NOT pushed):** the vLLM 0.26 `ParserEngine` (SemanticEvent -> streaming `DeltaMessage` + one-shot `ExtractedToolCallInformation`) with qwen3/seed_oss/kimi_k2 assembled parsers + `parser_manager` name->parser dispatch, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 9 scenarios, 1652/1652, RED-first 32 asserts, goldens byte-reproduced). **Serving-SSE dispatch swap LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-SERVING`, NOT pushed):** the OpenAI chat streaming path routes engine-backed `--tool-call-parser` names through `parser_manager get_parser_engine` and drives `parse_delta`/`parse`, EXACT chunk-for-chunk vs vLLM 0.26 `chat_completion_stream_generator` (`test_openai_serving_chat_stream` 9 scenarios, 210/210, RED-first 6 CHECKs, goldens byte-reproduced); OFF by default, legacy seam byte-identical (`test_openai_serving` 421/421 unchanged). **5 more engine CONFIG FAMILIES LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS`, NOT pushed):** minimax_m2/glm47_moe/deepseek_v4/deepseek_v32/nemotron_v3 ported as additive `ParserEngineConfig` builders + regex arg-converters + `Glm47MoeParser` name-`.strip()`, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 19 scenarios, 3510/3510, RED-first 2 asserts; engine-core 586/586 + serving-SSE 210/210 unchanged). **LAST 2 CONFIG FAMILIES gemma4 + inkling LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS-2`, NOT pushed) — vLLM tool-parser family parity CLOSED:** added 4 default-inert assembly-core virtual seams (`preprocess_feed`, virtual `events_to_delta`/`single_pass_parse`/`reset`/`extract_reasoning`, `args_wrapper_keys`) + `gemma4_config`/`inkling_config` (custom key:value + JSON-span arg carvers) + `Gemma4Parser` (channel-injection + `thought\n`-strip) / `InklingParser` (args-key unwrap + trailing-text flush), field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 26 scenarios, 4526/4526, adds a non-streaming parse() gate, RED-first for all 4 new seams; engine-core 586/586 + serving-SSE 210/210 byte-identical, seams inert). **LIVE PER-STEP METRIC WIRING LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-METRICS-WIRE`, NOT pushed) — the oldest T0 metrics debt fully RETIRED:** `/metrics` now serves LIVE values, not the primed schema. `EngineCoreOutputs.scheduler_stats` (new `Scheduler::make_stats()`, `scheduler.py:2399-2436`) + a stamped `timestamp`; `OutputProcessor::process_outputs` builds `IterationStats` (token counts, TTFT/ITL, finished-request breakdowns off `RequestState` timing, `stats.py:377-475`); the sync `LLMEngine::step()` folds both into the attached logger's `Record()` (`llm_engine.py:308-329`). Behavioural CPU gate `test_llm_engine.cpp` case 6 (44 assertions) drives the reference engine several steps: running/waiting gauges track the batch, prompt/generation counters == exact token counts, `request_success_total` counts finished reqs, TTFT/ITL/e2e/TPOT/iteration histograms observe the right sample counts; RED-first (14 asserts flip 0→correct with `Record` off). Additive/opt-in: null logger ⇒ byte-identical greedy stream; catalog gate `test_prometheus_metrics` 4/4/81 unchanged. **PER-REQUEST TIMING via EngineCoreEvents LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-RESPONSE-METRICS`, NOT pushed) — `SERVE-RESPONSE-METRICS` INVENTORIED→ACTIVE:** the scheduler records QUEUED/SCHEDULED/PREEMPTED `EngineCoreEvent`s 1:1 vLLM (`scheduler.py:2135/1003/1221`, gated on `log_stats_`), drained onto `EngineCoreOutput.events`; `OutputProcessor.update_from_events` folds them into `FinishedRequestStats.{queued,prefill,inference}_time` + `IterationStats.num_preempted_reqs`, so `vllm:request_{queue,prefill,inference}_time_seconds` + `vllm:num_preemptions_total` now carry real durations (the live-metrics wiring left them at 0). Gates: `test_scheduler` +1 (15 asserts, RED-first, real KV-exhaustion preemption) + `test_llm_engine` +1 (26 asserts, RED-first: 5 flip 0→positive; inference=prefill+decode, prefill≤inference≤e2e); no-logger path byte-identical. **CHAT-FORM `/tokenize` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-CHAT-TOKENIZE`, NOT pushed):** `/tokenize` accepts BOTH arms of the `TokenizeRequest` union — the chat form (`messages`+flags) renders through the SAME model chat template `create_chat_completion` uses (`chat_.prompt_fn()`), applies `check_generation_prompt` (→400), tokenizes with the chat-form `add_special_tokens` default False, returns `{count,max_model_len,tokens,token_strs?}` identically to vLLM 0.26; exact-gated tokens == render→Encode, RED-first (`test_openai_api_server` 27/27/337); raw-`prompt` form byte-identical. **JSON-SCHEMA ARG-TYPE COERCION LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-ARG-COERCION`, NOT pushed):** `_fix_arg_types`/`_streamable_string_keys`/`find_tool_properties` (parser_engine.py:227,269,365,348) ported over the already-ported `extract_types_from_schema`/`coerce_to_schema_type` helpers — `ParserTool` carries the function `parameters` schema (threaded from `serving_chat.cpp` `ToParserRequest`), so a request whose tools declare typed params has its assembled `tool_calls[].function.arguments` coerced to int/number/bool/string/array/null in BOTH streaming (`parse_delta`) and one-shot (`extract_tool_calls`/`parse`), 1:1 with vLLM (priority null>int>number>bool>object>array>string, uncoercible values left as-is); no-schema/absent-tools = identity byte-identical. Gate `test_parser_engine_assembly` 30 scenarios, 5038/5038 (scenarios 27-30: qwen3 typed-schema whole+char, schema-mismatch, kimi JSON-native `"5"`->int), RED-first 38 asserts (`"5"`-vs-`5`); engine-core 586/586 + serving-SSE 210/210 byte-identical. **`/tokenizer_info` (`SERVE-UTILITY-ENDPOINTS`) + `/abort_requests` (`SERVE-ADMIN`, INVENTORIED→ACTIVE) LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `GET /tokenizer_info` gated behind a `set_tokenizer_info_enabled` flag mirroring vLLM's `enable_tokenizer_info_endpoint` (off→route absent→404) surfaces the `tokenizer_config.json`-equivalent fields our byte-level/SentencePiece BPE tokenizer can GENUINELY back (`tokenizer_class`/`model_max_length`/`vocab_size`/`bos`/`eos`/`added_tokens_decoder`) and OMITS (never fabricates) the named gaps — the raw `chat_template` string (lives in the ChatPromptFn seam), the HF `init_kwargs`, and added-token `normalized`/`single_word`; `POST /abort_requests` (dev/rlhf admin router) parses `{request_ids}` and aborts the listed ids via an injected engine-abort callback (`AsyncLLM::abort`), empty→"abort all", returning `{"status":"aborted","aborted":N}` (malformed→400 `{"detail":...}`, failure→500 `{"error":...}`) — all shapes 1:1 vLLM. Both additive/opt-in (route only when backing attached), RED-first over a real socket (404 off → 200 on) + behaviour (abort tears down an in-flight AsyncLLM request); `test_openai_api_server` 31/31/379, existing conformance 23/23 + serving 40/40 byte-identical. **PRODUCTION `main.cpp` ENDPOINT WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary previously called NONE of the opt-in endpoint setters — a whole family of unit-tested endpoints was DARK in the actual server. A single shared seam `ConfigureUtilityEndpoints` (exercised by both `main.cpp` and the gate over a real socket) now lights, mirroring vLLM 0.26's per-endpoint default gating: `/tokenize`+`/detokenize` (on by default when a tokenizer exists), `/tokenizer_info` (new `--enable-tokenizer-info-endpoint`, mirrors `cli_args.py:140`), and `/abort_requests` (new `--enable-server-dev-mode`, mirrors `envs.VLLM_SERVER_DEV_MODE` at `api_server.py:238`) wired to the LIVE `AsyncLLM` with an exact before/after abort delta-count. `/metrics` + `/reset_prefix_cache` stay NAMED-BLOCKED (the AsyncLLM frontend exposes no live `PrometheusStatLogger` and no thread-safe prefix-cache reset RPC — attaching either would be a fabricated wiring). Core routes byte-identical; `test_openai_api_server` 32/32 (420, RED-first no-seam→404), conformance 23/23 + serving 40/40 unchanged; clean CPU `-Werror` incl. the `server` binary. RESIDUAL (honest): the chat/completion RESPONSE-BODY timing surface + CLI validation; AsyncLLM production-serving metric wiring (`/metrics` backing); config-gated metric families (spec-decode/kv-connector/mm/LoRA); `chat_template_kwargs`/`continue_final_message` full render passthrough; `/abort_requests` empty-list "abort ALL" enumeration; live `/reset_prefix_cache` backing | `PARTIAL` | response-body timing; AsyncLLM serving-path metric wiring + config-gated families; then DONE | | 9 | `ROAD-V1-C9` | Mechanical recurring upstream sync | [upstream sync](upstream-sync.md), [v0.25 audit](sync/2026-07-12-702f481.md), [porting inventory](porting-inventory.md) | v0.25.0 target `702f481` enumerated/classified: 145 post-pin commits, 94 inventory and 51 ignore, no unequivalent PORT-NOW runtime delta in the implemented T0 slice. The executable DGX oracle is validated/active at the target; the porting pin remains `e24d1b24` pending target goldens/behavior/model re-verification | `PARTIAL` | refresh exact performance denominators and target goldens/tests, then advance the parity pin | -| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **RPi5/A76 R2-R3 GREEN; `KERNEL-CPU-A76-Q8-DOT` SPIKED:** QEMU-built artifact is token/fixture exact on Pi; portable Q8 dot is 20.10% of user cycles and exact-order C++ SDOT vs scheduled AAPCS64 is the active gate. **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | +| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **RPi5/A76 R4-R5 GREEN; `KERNEL-CPU-A76-Q8-DOT` GATING:** QEMU-built artifact is token/fixture exact on Pi; scheduled AAPCS64 beats compiler SDOT 3.66-5.08% on M1/T1 and M128, lowers 64-token Qwen TTFT 1.55% and E2E 0.13%. A76+DotProd auto-selects assembly; M1/T4 (−2.43%), memory/concurrency and same-file llama.cpp floor remain open. **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | | 10a | `ROAD-V1-D1-CUDA` | **CUDA-arch derive-and-ship + community testing matrix (user-directed 2026-07-27):** the arch-additivity framework already makes a new CUDA device a near-verbatim add, so DERIVE-AND-SHIP the fast-path kernels rather than scope-and-wait — port 1:1 from vLLM (Ampere FA2/Marlin/AllSpark/scaled-mm-C2x; Hopper FA3/C3x-Sm90/Machete/MoE; datacenter-Blackwell C3x-Sm100a-tcgen05/MLA/fp4), and the fp16/non-tensor-core bodies from llama.cpp for the arches vLLM DROPS (Pascal/Volta/Turing = MORE than vLLM), build-verify (compile + `cuobjdump` SASS), and SHIP LABELED with a 3-state signal: RUNTIME-VERIFIED / DERIVED+BUILD-VERIFIED (testing-welcome) / NOT-YET-BUILDABLE (a build is never a runtime claim). Runtime-gate boards: **AGX Orin (sm_87, reachable)** + **NVIDIA Thor (Blackwell, reachable, SM to confirm on-box)** — the first two non-GB10 runtime proofs, spanning Ampere + Blackwell. DeepGEMM (runtime JIT) is the one honest NOT-YET-BUILDABLE. | [backend matrix](backend-matrix.md), [Ampere](specs/cuda-arch-ampere-fastpath.md), [Hopper+DC](specs/cuda-arch-datacenter-fastpath.md), [breadth](specs/cuda-arch-breadth-fp16.md) | 3 committed spikes (2026-07-27); backend rows moved INVENTORIED/build-only -> SPIKE; **FIRST NON-GB10 RUNTIME PROOF: Thor sm_110 portable bf16 path RUNTIME-VERIFIED (token-exact Llama-3.2-1B, `CLAIM-CUDA-SM110-RUNTIME`)**; **FIRST DERIVE-AND-SHIP BRICK: WA-1 FA2 Ampere enablement DERIVED+BUILD-VERIFIED (fa2 cell widened to 8.x; sm_87/sm_80 FA2 cubins, sm_121a SACRED 16/16 unchanged, no Ampere board ran it)**; **Turing sm_75 W1 BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-TURING-SM75`): the bf16-WMMA TU guard landed, single-arch `75` `-Werror` 0-warn + `cuobjdump` real `sm_75` cubin, sm_121a byte-identical — DERIVED+BUILD-VERIFIED (testing-welcome), no board ran it**; **DC1 datacenter-Blackwell sm_100a NVFP4 tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-NVFP4`): faithful 1:1 port of vLLM `Fp4GemmSm100` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05) as a NEW dedicated `cutlass-nvfp4-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror`-equiv 0-warn + `cuobjdump` real `sm_100a` cubin, RED (HEAD 100a fp4 DISABLED), sm_121a neutral — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; native `fp4-mma` stays sm_12x-only (mma.sync≠tcgen05)**; **DC2 Hopper sm_90a CUTLASS C3x FP8 scaled-mm BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM90-C3X`): faithful 1:1 port of vLLM `cutlass_3x_gemm_sm90_fp8` (`ArchTag=Sm90`+`KernelTmaWarpSpecialized*FP8FastAccum`→wgmma/TMA) as a NEW dedicated `scaledmm-c3x-sm90` FEATURE-TABLE cell (90a-only), single-arch `90a` `-Werror` 0-warn + `cuobjdump` real `sm_90a` cubin (ptxas C7510 names `wgmma.mma_async`), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no Hopper board ran it; sm90 int8/blockwise + sm100 C3x legs are residual bricks**; **DC3 datacenter-Blackwell sm_100a CUTLASS C3x FP8 scaled-mm tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-C3X`): the intersection of DC1's tcgen05 arch + DC2's C3x fp8 kernel — a faithful 1:1 port of vLLM `cutlass_3x_gemm_sm100_fp8` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05, `sm100_fp8_config_{default,M256,M64}`, 2SM `<_2,_2,_1>` default) as a NEW dedicated `scaledmm-c3x-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror all-warnings` 0-warn EXIT=0 + `cuobjdump` real `sm_100a` cubin (`Sm100TmaUmma`/`SM100_MMA_F8F6F4_2x1SM`/`TMEM` symbols, `LDTM`/`tmem` SASS), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8`/`cutlass-nvfp4`/`cutlass-nvfp4-sm100` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; sm100 int8/blockwise + MoE/MXFP4/MLA legs are residual bricks**; Volta/Pascal need CUDA<13 | ACTIVE | **SECOND NON-GB10 RUNTIME PROOF: Orin sm_87 portable bf16 SYNC path RUNTIME-VERIFIED 2026-07-28 (`CLAIM-CUDA-ORIN-SM87-RUNTIME`)** — built portable-only for sm_87 on a real Jetson AGX Orin (CUDA-13 container BLOCKED by the 12.6 driver → l4t-jetpack r36.4.0 nvcc 12.6 + g++-13; all fast paths EMPTY for [87]), and `test_llama_paged_engine` (Llama-3.2-1B bf16) is **13/16 strict token-exact vs the vLLM 0.25.0 oracle, 16/16 near-tie gate, 0 divergent** on the sm_87 GPU (exceeds Thor's 12/16), + test_cuda_backend/test_cuda_ops 461 assertions of real on-device kernels; ONE honest sm_87 bug found — the DEFAULT async runner crashes (`cudaFree illegal memory access`), SYNC path (`VT_ASYNC_RUNNER=0`) is the verified one (unblock item). Residual Orin W-plan: fix async runner -> GGUF leg -> FA2 (needs cutlass) -> Marlin-int4 -> vs llama.cpp; Thor fp8/fp4 fast paths once SSH-reachable | | 10b | `ROAD-V1-D1-GDN-AOT` | **Per-arch Triton-AOT GDN packed-decode cubins (user-directed 2026-07-28):** the vendored Triton-AOT GDN fast-path cubins — the MEASURED codegen-win decode (Triton REG:205/0-spill vs hand-CUDA REG:255+STACK:48 spills) plus the delta_h/chunk_o/kkt/tril/wu FLA set — existed for **`sm_121a` ONLY**, so GDN-hybrid models (Qwen3.6) ran GDN decode correct but on the SPILLING hand kernel on every other arch (all cross-family arch builds ship `-DVLLM_CPP_TRITON=OFF`). Regenerate the full GDN AOT set per-arch via the sanctioned `-DVLLM_CPP_TRITON_REGEN=ON` pipeline and vendor each `sm_XX/` tree so a `-DVLLM_CPP_TRITON=ON` build selects the non-spilling FLA path additively (build-time cubin selection = `sm_${VLLM_CPP_CUDA_ARCHITECTURES}`, no code change). | [kernel matrix](kernel-matrix.md), [backend matrix](backend-matrix.md) | [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `DONE (DERIVED+BUILD-VERIFIED)` | **LANDED 2026-07-28 (`CLAIM-TRITON-AOT-PER-ARCH`, base `308c312a`):** full GDN AOT set regenerated + vendored for `sm_80/86/89/90a/100a` on dgx GB10 (Triton 3.6.0 / ptxas 12.8, cross-compiled — no target board needed; 57 artifacts + MANIFEST per arch matching the `sm_121a` fileset), `cuobjdump` proves real per-target SASS (`sm=80/86/89/90/100`, decode REG 209–217/0-spill, under the hand-CUDA spill floor); builder-path configure selects + integrity-verifies each tree; `check-triton-aot-drift.sh` rc=0 across all six trees; `sm_121a` byte-untouched (SACRED 27B/35B gate structurally unchanged). **HONEST: DERIVED+BUILD-VERIFIED — no non-`sm_121` board runs a GDN model here, so decode parity on these arches is build-verified, NOT runtime-measured.** Residual: runtime GDN-model verification on Ampere/Ada/Hopper/DC-Blackwell silicon when reachable | | 11 | `ROAD-V1-D2` | Tensor/multi-GPU and MoE sequence parallelism | [engine matrix](engine-matrix.md), [coverage view §3](feature-matrix.md#3-parallelism--scale-out) | TP spec written; v0.25.0 non-DP MoE sequence-parallel path is inventoried as `PAR-SEQUENCE-MOE` but unspiked | `READY` | acquire 2-GPU target and claim Phase 0 mock/ABI; its execution trace determines whether sequence parallel is part of the first performance slice | diff --git a/.agents/specs/cpu-a76-q8-dot.md b/.agents/specs/cpu-a76-q8-dot.md index b38153a98..477c0f81e 100644 --- a/.agents/specs/cpu-a76-q8-dot.md +++ b/.agents/specs/cpu-a76-q8-dot.md @@ -1,4 +1,4 @@ -# KERNEL-CPU-A76-Q8-DOT spike +# KERNEL-CPU-A76-Q8-DOT Date: 2026-08-06 @@ -7,11 +7,11 @@ Row: `KERNEL-CPU-A76-Q8-DOT` Claim: `CLAIM-KERNEL-CPU-A76-Q8-DOT`, stacked branch `row/KERNEL-CPU-A76-Q8-DOT` -State: `SPIKE` +State: `GATING` ## Scope -Optimize the Q8_0 weight by Q8_0 activation dot used by +Optimizes the Q8_0 weight by Q8_0 activation dot used by `vt::MatmulBTQuant` on Raspberry Pi 5's Cortex-A76. The row owns a portable fallback, a compiler-generated Arm DotProd/SDOT implementation, a scheduled AAPCS64 implementation, runtime dispatch, same-binary selection, focused tests, @@ -59,11 +59,11 @@ proven, but the assembly schedule is selected only for Cortex-A76. | Source | Local destination | Decision | |---|---|---| | llama portable `quants.c:400` | existing `cpu_quant_dot.cpp` | permanent fallback and exact oracle | -| llama Arm DotProd structure | new `cpu_quant_dot_sdot.cpp` | ACLE `vdotq_s32`, exact per-block float order | -| measured A76 schedule | new `cpu_quant_dot_a76.S` | original implementation, AAPCS64, no stack spill in core loop | +| llama Arm DotProd structure | `cpu_quant_dot_sdot.cpp` | ACLE `vdotq_s32`, exact per-block float order | +| measured A76 schedule | `cpu_quant_dot_a76.S` | original implementation, AAPCS64 leaf hot path, no stack traffic | | Linux HWCAP/MIDR selection | new C++ TU plus `include/vt/quant.h` | ASIMDDP gate; A76-only assembly; env A/B control | | quant GEMM dispatch | `cpu_quant_dot.cpp` | selected function replaces only Q8_0 row | -| benchmark variants | `examples/cpu_kernel_bench/main.cpp` | `portable`, `sdot`, `a76-asm`, `auto` | +| benchmark variants | `examples/cpu_kernel_bench/main.cpp` | `portable`, `sdot`, `a76-asm`, `auto`; exact same-binary A/B | The optimized implementations return exactly the portable function signature. They perform both 16-byte SDOTs per Q8 block, horizontally add the four i32 @@ -136,26 +136,54 @@ gate; this row only closes the proven Q8 assembly/compiler gap. | W | Work | Exit | |---|---|---| | W0 | spike, row and claim | record checkers green, committed before code | -| W1 | selection seam + explicit benchmark variants | x86 fallback and QEMU smoke green | -| W2 | exact-order C++ SDOT | disassembly contains SDOT; fixture/model exact | -| W3 | scheduled AAPCS64 kernel | ABI/disassembly checks and direct tests green | -| W4 | interleaved Pi PMU A/B | assembly beats C++ in cycles and wall time | -| W5 | recursive Qwen A/B and checkpoint | no output/enclosing regression; docs current | +| W1 | **GREEN** selection seam + explicit benchmark variants | x86 fallback and QEMU smoke green | +| W2 | **GREEN** exact-order C++ SDOT | disassembly contains SDOT; fixture/model exact | +| W3 | **GREEN** scheduled AAPCS64 kernel | ABI/disassembly checks and direct tests green | +| W4 | **GREEN with T4 residual** interleaved Pi PMU A/B | 3.66-5.08% win on M1/T1 and M128; M1/T4 −2.43% | +| W5 | **GREEN** recursive Qwen A/B and checkpoint | exact output; TTFT improves; E2E tie; docs current | +| W6 | Whole-system/thread/competitor exhaustion | M1/T4 partition, BF16 GEMM, memory/concurrency and llama.cpp floor | + +## R4-R5 binding result + +The locally QEMU-built GCC 13.3 binary passed 20/20 focused cases and 150,258 +assertions, including explicit SDOT/assembly execution with random, unaligned, +zero, signed-extreme, scale-edge, ragged and invalid-`nrc` inputs. The physical +Pi retained exact fixture checksums for portable, compiler and assembly arms. + +Against compiler SDOT, assembly improves wall time by 3.66% for M=1/T1, +5.08% for M=128/T1 and 3.69% for M=128/T4. Cycles fall 3.17-4.61% and retired +instructions fall 9.74-10.24%. The compiler loop has a 48-byte frame and two +dependent SDOTs per block; the assembly leaf overlaps two independent block +chains without changing floating accumulation order. M=1/T4 is an honest +negative, 2.43% slower despite 8.77% fewer instructions, and remains a W6 +thread-partition gate. + +All three 64-token Pi arms are byte-identical to the x86 golden. Median +assembly versus compiler SDOT is 1.55% lower TTFT, neutral TPOT (0.05% lower) +and tied/slightly lower E2E (0.13%). Against the prior portable production +arm it lowers TTFT 33.40% and E2E 2.67%. `auto` therefore selects assembly +only on Cortex-A76+DotProd; every other CPU retains the portable path. The +immutable commands, hashes, disassembly and raw-file index are in +[the R5 evidence](../../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). + +W6 remains open for the M=1/T4 scheduler interaction, BF16-GEMM optimization, +peak memory, concurrency and the same-file llama.cpp floor. Those gaps keep +this row `GATING`; they do not undo the demonstrated assembly/compiler win. ## Risks and decisions -- The compiler may already schedule the intrinsic loop optimally. In that case - this assembly candidate is negative and the row remains open or selects a - different proven compiler gap; an assembly file is not success by itself. +- GCC did not schedule the intrinsic loop optimally: it emitted a framed, + one-block loop with two dependent SDOTs. The independent two-block assembly + schedule closes that measured gap on three of four binding shapes. - Q8 blocks are 34 bytes, so scale and payload streams are not naturally 16-byte aligned. Loads must remain unaligned-safe, and speculative reads may not cross the allocated final block. - Hardware FP16 conversion must match `F16ToF32`, including signed zero, infinities and NaNs. If payload handling differs, retain software conversion for specials or dispatch those inputs to portable code. -- Unrolling may expose memory-level parallelism but increases register and I- - cache pressure. PMU plus recursive model A/B, not instruction count alone, - decides the retained schedule. -- The R3 model is BF16-GEMM dominated. A clear dot-kernel win may be diluted; - it is still accepted as the requested demonstrated assembly win only when no - enclosing regression exists, and remains opt-in until model wall time moves. +- Two-block unrolling improved M=1/T1 and both M=128 arms, but M=1/T4 regressed + because the enclosing threadpool dominates that small shard. The model A/B, + not instruction count alone, justifies the retained A76-only default. +- The R3 model remains BF16-GEMM dominated. The assembly win mainly moves + prefill/TTFT; decode TPOT is neutral, which selects BF16 GEMM and the + M=1/T4 partition as the next recursive optimization targets. diff --git a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md index 8769a3cb9..c12637653 100644 --- a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md +++ b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md @@ -221,8 +221,8 @@ matching llama.cpp quant/repack cases for any borrowed layout or kernel: | R1 | **CPU-GATED** general CPU kernel/PMU harness | existing quant bench | warning-clean build; JSON/CLI/timer/counter contract; 1/4-thread runs | | R2 | **GREEN** QEMU-built portable Pi bring-up and x86 goldens | R0 | exact hash, load, 16/16 tokens, operation fixtures | | R3 | **GREEN** Qwen trace + recursive scope profiling | R1-R2 | reached-loop inventory and binding baseline below | -| R4 | A76 C++/NEON/SDOT providers | R3 ranked evidence | op correctness + causal metric win + no enclosing regression | -| R5 | A76 assembly candidates | R4 plateau + proven compiler gap | ABI/disassembly/correctness + recursive A/B | +| R4 | **GREEN** A76 C++/NEON/SDOT provider | R3 ranked evidence | exact operation/model output; 2.4x scalar kernel speedup | +| R5 | **GREEN with named T4 residual** A76 assembly candidate | R4 plateau + proven compiler gap | 3.66-5.08% binding kernel win; recursive model non-regressing | | R6 | Whole-system/thread/serving exhaustion | accepted R4/R5 stack | all lever dispositions, <1% residual model, llama.cpp floor | R1-R3 are the first implementation checkpoint. R4/R5 split into separate @@ -250,9 +250,27 @@ A zero-loss, low-overhead `cycles:u` profile of a 64-token model run ranks the reached loops: BF16 `Bt16Neon` 57.76%, portable `VecDotQ8_0Q8_0` 20.10%, thread-ready 6.45% and `F16ToF32` 4.87%. The Q8 dot is selected for R4/R5: the Pi has DotProd but the only existing Arm quant fast path requires i8mm, so -the real model currently executes the scalar portable dot. The next checkpoint -must compare portable, compiler-generated exact-order SDOT and AAPCS64 assembly -in one binary before any default dispatch changes. +the R4-R5 checkpoint therefore compared portable, compiler-generated +exact-order SDOT and AAPCS64 assembly in one binary before changing dispatch. + +## R4-R5 binding result + +The A76 Q8 row is now `GATING`, with its requested assembly/compiler gap +proven. The locally QEMU-built binary passed 20/20 focused cases and 150,258 +assertions. On the physical Pi, all operation checksums and all 64 Qwen tokens +match their portable/x86 goldens. The scheduled two-block AAPCS64 leaf beats +GCC's ACLE SDOT loop 3.66% on M=1/T1, 5.08% on M=128/T1 and 3.69% on +M=128/T4, while retiring about 10% fewer instructions. GCC emits a 48-byte +frame and two dependent SDOTs per block; the valid assembly hot path is a +stack-free leaf with two independent block chains. + +The recursive model gate is non-regressing: assembly versus compiler SDOT has +1.55% lower median TTFT, neutral TPOT and 0.13% lower E2E. Against the +portable arm it lowers E2E 2.67%. `auto` selects the assembly only on +Cortex-A76+DotProd. M=1/T4 remains a measured negative at −2.43%, so R6 owns +that thread-partition interaction plus BF16 GEMM, peak memory, concurrency and +the same-file Pi llama.cpp floor. Full provenance and raw hashes are in the +[R5 evidence](../../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). ## Risks and decisions diff --git a/.agents/state.md b/.agents/state.md index 70143dfe9..edffdbabc 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39380,3 +39380,61 @@ operation checksums and model tokens, a proven ABI/disassembly contract, and no enclosing Qwen regression. QEMU remains build/smoke-only and source will not be compiled on the Pi. No optimized implementation or speed claim exists at this spike checkpoint. + +## 2026-08-06T19:45 - KERNEL-CPU-A76-Q8-DOT R4-R5 assembly win + + +`CLAIM-KERNEL-CPU-A76-Q8-DOT` closes W1-W5 and moves the kernel row from +`SPIKE` to `GATING`. ARM64 was never compiled on the Pi: local buildx/QEMU +with Ubuntu 24.04/GCC 13.3 built the assembly TU, ran `test_ops_quant_dot` +20/20 (150,258 assertions) and executed an explicit assembly smoke before +export. Final binary SHA-256 values are `vllm-bench` +`9eb57cf3760eaade9dcef03dda1648556577c44199369ad38bf42083efbc70a9` +and `vllm-cpu-kernel-bench` +`a94dad30411651901e4f6ed8aaf14efb735e09f0412bb8f8788873fdfd7a6818`. + +The implementation keeps the portable Q8_0 dot as the universal fallback, +adds a GCC/ACLE exact-order SDOT control, and adds an AAPCS64 two-block leaf. +Linux HWCAP gates DotProd; MIDR implementer `0x41`, part `0xd0b` gates the +automatic assembly selection. Only `cpu_quant_dot_sdot.cpp` receives +`-march=armv8.2-a+dotprod+fp16`. Explicit benchmark overrides remain available +through `VT_CPU_Q8_DOT=portable|sdot|a76-asm`; `auto` selects assembly only on +Cortex-A76+DotProd and otherwise stays portable. + +Final disassembly proves the compiler gap. GCC's 216-byte `VecDotQ8Sdot` has a +48-byte stack frame and a one-block loop whose two adjacent SDOTs feed one +dependent accumulator. The 276-byte assembly valid path is a leaf with no +stack traffic, overlaps two independent `v4`/`v20` block chains, post-indexes +the 68-byte Q8 pair, uses only caller-saved registers and retains original +per-block f32 accumulation order. Invalid K/`nrc` tail-branches to the C++ +contract checker. + +Physical-Pi evidence is one same-binary interleaved series per shape, seven +outer repetitions per arm, ondemand up to 2.4 GHz and `throttled=0x0`. +Assembly versus compiler SDOT medians: M=1/T1 614,414.75 vs 637,738.5 ns +(**+3.66%**), cycles 1,469,048 vs 1,517,121 and instructions 3,664,537.5 vs +4,076,185.5; M=128/T1 75,537,409 vs 79,579,829 ns (**+5.08%**); M=128/T4 +19,887,768 vs 20,649,426 ns (**+3.69%**). Retired instructions fall +9.74-10.24%. The named negative is M=1/T4, 480,210 vs 468,798.25 ns +(**-2.43%**) and 4.32% more cycles despite 8.77% fewer instructions, +selecting the threadpool partition as W6 +rather than hiding it. + +All portable/compiler/assembly fixture arms are checksum-identical. The three +64-token Qwen repetitions for each arm are also byte-identical to the x86 +golden SHA-256 +`0ec98eabb23e4148d540fcf79a2fe61678fb90fe462cdf28134af7a42fe6a826`. +Median assembly versus compiler SDOT: TTFT 1,307.27 vs 1,327.91 ms +(**1.55% lower**), TPOT 367.67 vs 367.85 ms (0.05% lower, neutral), E2E +24,470.20 vs 24,502.52 ms (**0.13% lower**). Against portable, assembly +lowers TTFT 33.40%, E2E 2.67% and raises output throughput 2.75%. A final +`auto` 16-token run selected assembly, matched the original x86 golden and +reported TTFT 1,305.62 ms / TPOT 362.63 ms / E2E 6,745.08 ms without +throttling. + +Binding commands, all raw-file SHA-256 values, the four shape tables and +disassembly excerpts are indexed at +`docs/bench-evidence/rpi5-a76-q8-dot-20260806.md`. W6 remains open for the +M=1/T4 scheduler interaction, the still-dominant BF16 GEMM, peak memory, +concurrency and the same-file Pi llama.cpp floor. No Pi competitor-parity +claim is made. diff --git a/CMakeLists.txt b/CMakeLists.txt index eced89f65..89e48c610 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -748,6 +748,7 @@ add_library(vllm STATIC src/vt/cpu/cpu_quant_dequant.cpp src/vt/cpu/cpu_quant_act.cpp src/vt/cpu/cpu_quant_dot.cpp + src/vt/cpu/cpu_quant_dot_sdot.cpp src/vt/cpu/cpu_quant_dot_arm.cpp src/vt/cpu/cpu_quant_repack.cpp src/vt/cpu/cpu_quant_repack_arm.cpp @@ -769,6 +770,17 @@ add_library(vllm::vllm ALIAS vllm) # baseline, and the kernels are additionally runtime-gated on HWCAP2_I8MM so the # binary still runs on non-i8mm aarch64 (portable tier serves those shapes). if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|arm64") + # KERNEL-CPU-A76-Q8-DOT is Linux-only because its dispatch uses HWCAP/MIDR + # and the hand-written file carries ELF assembler directives. Other arm64 + # platforms compile the generic stub and retain their existing portable tier. + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + enable_language(ASM) + target_sources(vllm PRIVATE src/vt/cpu/cpu_quant_dot_a76.S) + set_source_files_properties( + src/vt/cpu/cpu_quant_dot_sdot.cpp PROPERTIES + COMPILE_DEFINITIONS VT_CPU_A76_Q8_DOT=1 + COMPILE_OPTIONS "$<$:-march=armv8.2-a+dotprod+fp16>") + endif() # G6 mmla tier AND G7 repack tier (cpu_quant_repack_arm.cpp) both need i8mm. # The global -ffp-contract=off (top of this file) still applies on top of this # per-file -march, which is what keeps the repack gemm/gemv's non-fused diff --git a/docker/Dockerfile.arm64 b/docker/Dockerfile.arm64 index b513808a0..cd6851e55 100644 --- a/docker/Dockerfile.arm64 +++ b/docker/Dockerfile.arm64 @@ -1,10 +1,23 @@ ARG UBUNTU=24.04 FROM ubuntu:${UBUNTU} AS build ARG JOBS=4 -RUN apt-get update && apt-get install -y --no-install-recommends build-essential cmake ca-certificates && rm -rf /var/lib/apt/lists/* +ARG DEBIAN_FRONTEND=noninteractive +RUN apt-get update && apt-get install -y --no-install-recommends build-essential cmake ca-certificates python3 && rm -rf /var/lib/apt/lists/* WORKDIR /src -COPY . . -RUN --mount=type=cache,id=vllm-cpp-arm64-build,target=/build cmake -S . -B /build -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_SERVER=OFF && cmake --build /build -j${JOBS} --target vllm-bench vllm-cpu-kernel-bench && /build/examples/vllm-cpu-kernel-bench --m 1 --n 32 --k 32 --threads 1 --warmup 0 --iterations 1 --counters off --no-pin --format json && mkdir -p /out && cp /build/examples/vllm-bench /build/examples/vllm-cpu-kernel-bench /out/ +RUN --mount=type=bind,source=.,target=/src,readonly \ + --mount=type=cache,id=vllm-cpp-arm64-build,target=/build \ + cmake -S /src -B /build -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_SERVER=OFF \ + && cmake -E rm -f \ + /build/CMakeFiles/vllm.dir/src/vt/cpu/cpu_quant_dot.cpp.o \ + /build/CMakeFiles/vllm.dir/src/vt/cpu/cpu_quant_dot_sdot.cpp.o \ + /build/CMakeFiles/vllm.dir/src/vt/cpu/cpu_quant_dot_a76.S.o \ + /build/examples/CMakeFiles/vllm-cpu-kernel-bench.dir/cpu_kernel_bench/main.cpp.o \ + /build/tests/CMakeFiles/test_ops_quant_dot.dir/vt/test_ops_quant_dot.cpp.o \ + && cmake --build /build -j${JOBS} --target vllm-bench vllm-cpu-kernel-bench test_ops_quant_dot \ + && /build/tests/test_ops_quant_dot \ + && /build/examples/vllm-cpu-kernel-bench --dtype q8_0 --m 1 --n 32 --k 64 --threads 1 --variant a76-asm --warmup 0 --iterations 1 --counters off --no-pin --format json \ + && mkdir -p /out \ + && cp /build/examples/vllm-bench /build/examples/vllm-cpu-kernel-bench /out/ FROM scratch AS export COPY --from=build /out/vllm-bench / COPY --from=build /out/vllm-cpu-kernel-bench / diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 62112bf9d..090a0f557 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -8,7 +8,7 @@ | **vLLM** | Qwen3.6-35B-A3B NVFP4, GB10 | 0.93x to 1.03x: ahead at c4, worst c16 0.93x | identical | | **vLLM** | DeepSeek-V2-Lite (MLA), GB10 | 0.86x to 0.95x throughput, TTFT wins at c4/c8 | identical | | **vLLM** | Laguna-XS-2.1 NVFP4, GB10 | **parity+, 1.03x** (44.46 vs 43.10 tok/s, byte-exact, default config; bf16 weights now device-resident) | near-tie | -| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill **1.18x ahead**, decode tie, memory parity. RPi5/A76: **PENDING** | byte-identical on binding arm; Pi pending | +| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill **1.18x ahead**, decode tie, memory parity. RPi5/A76: assembly component gate **1.04-1.05x vs compiler SDOT**; same-file llama.cpp floor pending | byte-identical on both Arm lanes | | **MLX-LM** | Qwen3-0.6B, Apple M4 | 97.6% warm total, prefill ahead | near-tie | | **DwarfStar** | DeepSeek-V4-Flash GGUF, GB10 | **beats ds4, 1.144x** (18.69 vs 16.33 tok/s, byte-exact, default config) | n/a, GGUF peer | @@ -156,22 +156,27 @@ host mirror is freed once the device Marlin resident is built. Raspberry Pi 5 Cortex-A76 is a separate `GATING` arm. It has four cores, DotProd and no i8mm, so the binding 20-core Arm result below does not transfer. -The R2-R3 portable baseline uses the exact Q8_K_XL SHA-256 in the -[RPi5 spike](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md) and a local -buildx/QEMU ARM64 build; QEMU timing is non-binding. The exported artifact ran -unthrottled at 2.4 GHz on the Pi and matched the x86 golden 16/16 tokens. Q8_0 -M=1/N=3072/K=2048 measured 1,554,115 ns at one thread and 742,585 ns at four; -M=128 measured 197,061,735 ns and 49,890,756 ns respectively. Exact fixture -checksums held in every arm. The 16-token model arm measured TTFT 1,961.99 ms, -TPOT/ITL 366.91 ms and output throughput 2.14 tok/s. A zero-loss 64-token -`cycles:u` trace attributes 57.76% to BF16 GEMM and 20.10% to the portable Q8 -dot. These numbers bind the portable denominator only. SDOT/assembly A/B, -three interleaved full-model repetitions, peak memory and same-file llama.cpp -comparison remain `PENDING`. - -The selected follow-on is `KERNEL-CPU-A76-Q8-DOT`. Its binding performance -gate compares portable, exact-order compiler SDOT and scheduled AAPCS64 in one -QEMU-built binary on the physical Pi. No assembly speedup is claimed at spike. +ARM64 artifacts are built and tested locally with buildx/QEMU, then copied to +the Pi for execution only. The exact Q8_K_XL model and all output/checksum +hashes are fixed in the [campaign spec](../.agents/specs/rpi5-cortex-a76-cpu-optimization.md). + +`KERNEL-CPU-A76-Q8-DOT` now closes the compiler-scheduling hypothesis. In one +GCC 13.3 binary on an unthrottled Pi, the scheduled AAPCS64 leaf beats +the exact-order compiler SDOT loop by **3.66%** on M=1/T1, **5.08%** on +M=128/T1 and **3.69%** on M=128/T4, with 9.74-10.24% fewer retired +instructions. M=1/T4 is the named residual, **2.43% slower**, despite fewer +instructions, so its thread partition remains open. + +The recursive 64-token Qwen gate is byte-identical across x86, portable, +compiler SDOT and assembly. Median assembly versus compiler SDOT is **1.55% +lower TTFT** (1,307.27 vs 1,327.91 ms), neutral TPOT (367.67 vs 367.85 ms), +and **0.13% lower E2E** (24,470.20 vs 24,502.52 ms). Against portable it +lowers TTFT 33.40% and E2E 2.67%. Cortex-A76+DotProd therefore selects the +assembly arm by default; every other CPU retains portable dispatch. Exact +commands, binary/raw-file hashes, all four kernel arms and the disassembly are +in the [immutable evidence index](bench-evidence/rpi5-a76-q8-dot-20260806.md). +Peak memory, concurrent serving and the same-file Pi llama.cpp comparison stay +`PENDING`; no Pi competitor-parity claim is made. Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 5c6fb2f98..0e61ef361 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -76,6 +76,7 @@ portable/reference path. In normal operation leave them unset. | `VT_FA2_DECODE_4B` | on (CUDA, Qwen3.5-4B) | The portable paged decode attention instead of the ratio-4 vendored FA2 path; the 27B and 35B selectors are unchanged | | `VT_FA2_DECODE_GQA_SWAP` | **on** (CUDA, Qwen3-dense d128 decode) | Applies vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization in the d128 varlen launcher (launches (batch, kv_heads) not (batch, hq), packing the ngroups query heads into seqlen_q); `=0` opts out and restores the byte-identical plain-varlen reduction. Flipped default-on 2026-08-06 (`KERNEL-FA2-GQA-SWAP-FLIP`): SACRED Qwen3-0.6B/4B 16/16 token-identical, binding q3mxfp4 tput c2-c8 0.911/0.919/0.913→0.922/0.930/0.942 with no c1 regression, mem 2.614x less | | `VT_CPU_REF` | off | Set on to force the portable reference path (dequantize-everything oracle), the standard "is this a kernel bug?" bisect switch | +| `VT_CPU_Q8_DOT` | `auto` | Selects the CPU Q8_0 x Q8_0 inner dot: `portable`, compiler-generated `sdot`, or `a76-asm`. `auto` selects the gated assembly schedule only on Cortex-A76 with DotProd and otherwise stays portable. Unsupported ISA selections safely fall back; `vllm-cpu-kernel-bench --variant` fails loudly instead | | `VT_DFLASH_PAGED` | on (CUDA, DFlash spec-decode) | The materialized `[context;block]` draft forward instead of the fixed-capacity paged draft-KV store read through `vt::DFlashPagedBlockAttention` (bit-identical; only the DFlash single-request propose path) | | `VT_DFLASH_GRAPH` | on (CUDA, DFlash spec-decode) | The eager paged draft step instead of the captured/replayed draft-step CUDA graph (replayed==eager bit-identical; only the DFlash single-request propose path) | | `VT_DFLASH_ATTN_BLOCK` | off (CUDA, DFlash spec-decode) | `=1` selects the D12/D13 block-per-(query,head) draft paged-attention kernel instead of the D14 default warp-scoped online-softmax kernel (same f32-softmax math within the bf16 envelope; the D14 warp kernel is ~3x faster and closed the ~2% speed residual; only the DFlash single-request propose path) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index cf9d69378..096dc24ae 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; A76 baseline green, Q8 SDOT/assembly spiked) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; A76 Q8 SDOT assembly gated and default) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index f4990038b..d668d0c5b 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 `GATING` | Binding: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi R2-R3 is 16/16 x86-token and fixture exact; Q8 dot is 20.10% of user cycles. `KERNEL-CPU-A76-Q8-DOT` is `SPIKE`: exact-order C++ SDOT vs AAPCS64 assembly A/B next; Pi llama.cpp floor open. Concurrency open | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 Q8 assembly `GATING` | Binding 20-core arm: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi: QEMU-built AAPCS64 leaf is output-exact and beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4; 64-token Qwen TTFT −1.55% and E2E −0.13% vs SDOT. M1/T4 is −2.43%, so thread partition, peak memory, concurrency and same-file Pi llama.cpp floor remain open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | diff --git a/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md b/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md new file mode 100644 index 000000000..9ebce2f6b --- /dev/null +++ b/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md @@ -0,0 +1,145 @@ +# Raspberry Pi 5 Cortex-A76 Q8 SDOT assembly, 2026-08-06 + +Immutable evidence index for `KERNEL-CPU-A76-Q8-DOT`. ARM64 artifacts were +built and tested locally with Docker buildx/QEMU, then copied to the Pi for +execution only. Nothing was compiled on the Pi. + +## Proven result + +The hand-written AAPCS64 loop is faster than GCC 13.3's ACLE `vdotq_s32` +loop on the physical Cortex-A76. It wins the binding M=1 single-core case and +both M=128 cases, preserves exact operation checksums, preserves all 64 model +tokens, lowers Qwen TTFT, and ties the compiler arm on end-to-end model time. +It is now the `auto` choice only on Cortex-A76 plus DotProd. + +One residual is retained rather than hidden: M=1 at four threads is 2.43% +slower than compiler SDOT despite retiring fewer instructions. That small +fixture is dominated by the enclosing threadpool schedule; the recursive +64-token model gate is neutral on TPOT and non-regressing end to end. The row +therefore remains `GATING`, with the M=1/T4 partition and the same-file +llama.cpp floor still open. + +## Provenance + +- Target: Raspberry Pi 5, four Cortex-A76 cores, MIDR `0x414fd0b1`, Arm + DotProd present, i8mm absent, ondemand clock up to 2.4 GHz and no throttling. +- Model: `unsloth/Qwen3.5-2B-GGUF`, file + `Qwen3.5-2B-UD-Q8_K_XL.gguf`, HF commit + `cc69d5651701145e5e14177cec01158109d37e5b`, SHA-256 + `a53988df91157d78acaf3c95e22db179d13f6236061bdb86576494dc99b1bc3b`. +- Compiler/build: Ubuntu 24.04, GCC 13.3, Release, local buildx builder + `pf-arm`, `linux/arm64`, Dockerfile `docker/Dockerfile.arm64`. QEMU ran the + focused test and explicit assembly smoke before export. +- Exported final binaries: `vllm-bench` + `9eb57cf3760eaade9dcef03dda1648556577c44199369ad38bf42083efbc70a9`; + `vllm-cpu-kernel-bench` + `a94dad30411651901e4f6ed8aaf14efb735e09f0412bb8f8788873fdfd7a6818`. +- Execution root: `rich@rpi5fan.lan:~/vllm-cpp-assembly`. Every accepted leg + reported `throttled=0x0`; checksums agreed within each series. + +## Kernel A/B + +Each file contains seven interleaved outer repetitions of portable, compiler +SDOT, and assembly in one binary. The table reports the median of each arm's +seven per-run medians. M=1 uses 15 timed samples per run; M=128 uses nine. +Counters are grouped `perf_event_open` user-space counts with running ratio 1. + +| Shape / threads | Portable ns | Compiler SDOT ns | Assembly ns | Assembly vs SDOT | SDOT cycles | Assembly cycles | SDOT instructions | Assembly instructions | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| M=1, T=1 | 1,558,076 | 637,738.5 | **614,414.75** | **3.66% faster** | 1,517,121 | **1,469,048** | 4,076,185.5 | **3,664,537.5** | +| M=1, T=4 | **423,474.625** | 468,798.25 | 480,210 | **2.43% slower** | **4,611,913.75** | 4,811,347.625 | 4,555,790.125 | **4,156,096** | +| M=128, T=1 | 197,324,228 | 79,579,829 | **75,537,409** | **5.08% faster** | 190,099,585 | **181,331,788** | 514,401,416 | **461,710,472** | +| M=128, T=4 | 49,896,520 | 20,649,426 | **19,887,768** | **3.69% faster** | 198,209,663 | **190,886,492** | 540,637,814 | **487,972,480** | + +The assembly reductions use the same output checksum as both controls: +M=1 `0xd6aec014c0050fda`, M=128 `0xa89baff1f3a4e360`. + +Raw JSONL SHA-256: + +| File under `~/vllm-cpp-assembly/evidence` | SHA-256 | +|---|---| +| `q8-dot-m1-t1-final-commit.jsonl` | `bc10143ab1e24cd29e6ec3e7139507eaded529fe54ec2a6c0b63d30ce20da1d6` | +| `q8-dot-m1-t4-final-commit.jsonl` | `577fc03d22470769fad1434e3bd482b9726a7958965f2f3f08f629b29d93b728` | +| `q8-dot-m128-t1-final-commit.jsonl` | `3de5c8ba268bff2da7efcea452dea145f534e2c797c9e3e6f313169ac7598b2f` | +| `q8-dot-m128-t4-final-commit.jsonl` | `d6e44db2607218d0e30ffeb22cdea75db7af665de93b81cc81592a6e164332bd` | + +The matching throttle/clock manifests are +`a7f30a9f04e7e38ff670e43e7275cb1db649a6c709f2493f82121ad4bc5d29c7`, +`21d535c169504b323f455aaa0610a009d0dcce006788a1b1f1af376cdd942695`, +`8295cdc02c9216a2fc34a2f6ab35a2b907fe25c1807752e2c7ce8b5d32512d1e` +and `41f4ef6c9e19509d2e03e8c9f76a130c8d9c6c48fd5f7d653e3e2d4734a8d383` +in the same shape order; every recorded throttle line is `throttled=0x0`. + +## Compiler gap and ABI audit + +Final `objdump -dC` gives GCC's `VecDotQ8Sdot` a 48-byte stack frame. Its +one-block loop issues two adjacent SDOTs into the same accumulator: + +```text +sdot v1.4s, v0.16b, v2.16b +... +sdot v1.4s, v0.16b, v2.16b +addv s1, v1.4s +``` + +The assembly valid path is a leaf with no stack traffic. Its two-block loop +uses independent `v4` and `v20` SDOT chains, interleaves unaligned-safe loads, +and post-indexes the 68-byte block pair while retaining the original scalar +floating-point block order. It uses only caller-saved integer/vector registers +and declares `armv8.2-a+dotprod+fp16`. Invalid `n`/`nrc` tail-branches to the +C++ contract checker. Final symbols are `VecDotQ8Sdot` at `0x88d380` (216 +bytes) and `vt_cpu_q8_dot_a76_asm` at `0x89e850` (276 bytes). + +## Recursive Qwen gate + +Three repetitions per variant, one request, generated input length 16, greedy +64-token output, four CPU workers. Every output file is byte-identical to the +x86-64 golden, SHA-256 +`0ec98eabb23e4148d540fcf79a2fe61678fb90fe462cdf28134af7a42fe6a826`. + +| Median axis | Portable | Compiler SDOT | Assembly | Assembly vs SDOT | Assembly vs portable | +|---|---:|---:|---:|---:|---:| +| TTFT (ms) | 1,962.85 | 1,327.91 | **1,307.27** | **1.55% lower** | **33.40% lower** | +| TPOT (ms) | 367.91 | 367.85 | **367.67** | 0.05% lower, neutral | 0.07% lower | +| E2E (ms) | 25,142.40 | 24,502.52 | **24,470.20** | **0.13% lower** | **2.67% lower** | +| Output throughput (tok/s) | 2.55 | 2.61 | **2.62** | 0.38% higher | **2.75% higher** | + +The final `auto` smoke selected assembly on the Pi, matched the 16-token x86 +golden SHA-256 +`684f55a32355c0ccb6ce9c987273981f077b9591a46db07aea68561eb6432966`, +and measured TTFT 1,305.62 ms, TPOT 362.63 ms, E2E 6,745.08 ms with no +throttling. + +## Reproduction + +Build and execute tests locally through QEMU: + +```sh +docker buildx build --builder pf-arm --platform linux/arm64 \ + --file docker/Dockerfile.arm64 --output type=local,dest=/tmp/vllm-arm64 . +``` + +For each `(M,threads)` and each `variant` in a randomized/interleaved order, +run a fresh process seven times. Use `--iterations 15` for M=1 and +`--iterations 9` for M=128: + +```sh +VLLM_CPP_CPU_THREADS=1 bin/vllm-cpu-kernel-bench \ + --dtype q8_0 --m 1 --n 3072 --k 2048 --threads 1 \ + --variant a76-asm --warmup 2 --iterations 15 --cache hot \ + --counters all --seed 0 --format json +``` + +The model command is: + +```sh +VLLM_CPP_CPU_THREADS=4 VT_CPU_Q8_DOT=a76-asm bin/vllm-bench \ + --model models/Qwen3.5-2B-UD-Q8_K_XL.gguf \ + --num-prompts 1 --input-len 16 --output-len 64 --concurrency 1 \ + --seed 0 --temperature 0 --output-token-ids evidence/out.json +``` + +Replace `a76-asm` with `sdot` and `portable`; verify `sha256sum` and +`vcgencmd get_throttled` before and after every arm. QEMU timings are not +performance evidence. The same-file llama.cpp comparison remains pending and +no competitor parity claim is made here. diff --git a/examples/cpu_kernel_bench/main.cpp b/examples/cpu_kernel_bench/main.cpp index a75f106d0..17061e9b6 100644 --- a/examples/cpu_kernel_bench/main.cpp +++ b/examples/cpu_kernel_bench/main.cpp @@ -85,7 +85,8 @@ T ParseInteger(std::string_view value, std::string_view flag) { void Usage(std::ostream& out) { out << "vllm-cpu-kernel-bench, PMU-backed CPU operation harness\n\n" << " --op matmul-bt-quant --dtype q8_0|q4_k|q6_k\n" - << " --m N --n N --k N --threads N --variant auto|portable|repacked|mmla\n" + << " --m N --n N --k N --threads N\n" + << " --variant auto|portable|sdot|a76-asm|repacked|mmla\n" << " --warmup N --iterations N --cache hot|l2|l3|stream\n" << " --counters auto|off|generic|a76|all --format text|json\n" << " --seed N --no-pin\n\n" @@ -150,7 +151,8 @@ Options ParseArgs(int argc, char** argv) { if (o.m <= 0 || o.n <= 0 || o.k <= 0) Fail("M, N and K must be positive"); if (o.threads <= 0 || o.threads > 512) Fail("--threads must be in [1,512]"); if (o.warmup < 0 || o.iterations <= 0) Fail("warmup must be >=0 and iterations >0"); - const std::array variants = {"auto", "portable", "repacked", "mmla"}; + const std::array variants = {"auto", "portable", "sdot", + "a76-asm", "repacked", "mmla"}; const std::array caches = {"hot", "l2", "l3", "stream"}; const std::array counters = {"auto", "off", "generic", "a76", "all"}; if (std::find(variants.begin(), variants.end(), o.variant) == variants.end()) @@ -552,6 +554,11 @@ class QuantMatmulFixture { repacked_ = true; } if (o.variant == "mmla" && !vt::cpu::QuantMmlaActive()) Fail("mmla is unavailable"); + if ((o.variant == "sdot" || o.variant == "a76-asm") && dtype_ != vt::DType::kQ8_0) + Fail("sdot and a76-asm require q8_0"); + if (o.variant == "sdot" && !vt::cpu::QuantQ8SdotActive()) Fail("sdot is unavailable"); + if (o.variant == "a76-asm" && vt::cpu::QuantQ8A76AsmVecDot() == nullptr) + Fail("a76-asm is unavailable"); at_ = vt::Tensor::Contiguous(activations_.data(), vt::DType::kF32, q_.device, {o.m, o.k}); bt_ = vt::Tensor::Contiguous(weights_.data(), vt::DType::kF32, q_.device, {o.n, o.k}); bt_.dtype = dtype_; @@ -851,10 +858,15 @@ int main(int argc, char** argv) { try { Options options = ParseArgs(argc, argv); SetEnv("VLLM_CPP_CPU_THREADS", std::to_string(options.threads)); - if (options.variant == "portable") { + if (options.variant == "portable" || options.variant == "sdot" || + options.variant == "a76-asm") { SetEnv("VT_CPU_QUANT_MMLA", "0"); SetEnv("VT_CPU_QUANT_REPACK", "0"); } + if (options.variant == "portable" || options.variant == "sdot" || + options.variant == "a76-asm") { + SetEnv("VT_CPU_Q8_DOT", options.variant); + } if (options.pin) PinToFirstCpus(options.threads); QuantMatmulFixture fixture(options); // Open before the warmup creates persistent workers. inherit=1 then makes diff --git a/include/vt/quant.h b/include/vt/quant.h index b3aace4bd..6956cf32f 100644 --- a/include/vt/quant.h +++ b/include/vt/quant.h @@ -73,6 +73,17 @@ FromFloatFn BlockFromFloat(DType dtype); // cpu_quant_dot.cpp. VecDotFn BlockVecDot(DType dtype); +// Q8_0 x Q8_0 DotProd variants for KERNEL-CPU-A76-Q8-DOT. The explicit +// getters are test/benchmark seams; SelectQuantQ8VecDot applies +// VT_CPU_Q8_DOT=auto|portable|sdot|a76-asm while retaining `portable` as the +// universal fallback. The assembly getter is ISA-safe on any DotProd core; +// QuantQ8A76AsmActive additionally reports whether the running CPU is A76. +VecDotFn QuantQ8SdotVecDot(); +VecDotFn QuantQ8A76AsmVecDot(); +VecDotFn SelectQuantQ8VecDot(VecDotFn portable); +bool QuantQ8SdotActive(); +bool QuantQ8A76AsmActive(); + // The Arm i8mm (mmla) `nrc == 2` `vec_dot` for a block WEIGHT dtype — QUANT- // GGUF-CIQ-GEMM work row G6 (cpu_quant_dot_arm.cpp). Non-null ONLY when the // process runs on i8mm-capable aarch64 (compile-time `__ARM_FEATURE_MATMUL_INT8` diff --git a/src/vt/cpu/cpu_quant_dot.cpp b/src/vt/cpu/cpu_quant_dot.cpp index 800775f14..a891c3c84 100644 --- a/src/vt/cpu/cpu_quant_dot.cpp +++ b/src/vt/cpu/cpu_quant_dot.cpp @@ -675,7 +675,8 @@ void VecDotMXFP4Q8_0(int n, float* s, size_t bs, const void* vx, size_t bx, VecDotFn BlockVecDot(DType dtype) { switch (dtype) { case DType::kQ4_0: return &VecDotQ4_0Q8_0; // quants.c:174 - case DType::kQ8_0: return &VecDotQ8_0Q8_0; // quants.c:400 + case DType::kQ8_0: + return SelectQuantQ8VecDot(&VecDotQ8_0Q8_0); // quants.c:400 + A76 tier case DType::kQ2_K: return &VecDotQ2_KQ8_K; // quants.c:514 case DType::kQ3_K: return &VecDotQ3_KQ8_K; // quants.c:566 case DType::kQ4_K: return &VecDotQ4_KQ8_K; // quants.c:645 diff --git a/src/vt/cpu/cpu_quant_dot_a76.S b/src/vt/cpu/cpu_quant_dot_a76.S new file mode 100644 index 000000000..c55b89ab3 --- /dev/null +++ b/src/vt/cpu/cpu_quant_dot_a76.S @@ -0,0 +1,94 @@ +// Scheduled Cortex-A76 Q8_0 x Q8_0 VecDotFn. AAPCS64 leaf, no stack frame on +// the valid hot path. Invalid arguments tail-branch to the C++ throw helper. +// Two blocks are kept in flight so load/SDOT/FP16-convert latency can overlap; +// the two scalar contributions are still added in original block order. + .arch armv8.2-a+dotprod+fp16 + .text + .p2align 4 + .global vt_cpu_q8_dot_a76_asm + .type vt_cpu_q8_dot_a76_asm, %function +vt_cpu_q8_dot_a76_asm: + .cfi_startproc + tst w0, #31 + b.ne .Lbad_args + cmp w7, #1 + b.ne .Lbad_args + mov x9, x1 + mov x10, x3 + mov x11, x5 + asr w2, w0, #5 + fmov s31, wzr + cmp w2, #0 + b.le .Ldone + cmp w2, #2 + b.lt .Ltail + +.Lpairs: + ldur q0, [x10, #2] + ldur q1, [x11, #2] + movi v4.4s, #0 + ldur q16, [x10, #36] + sdot v4.4s, v0.16b, v1.16b + ldur q17, [x11, #36] + movi v20.4s, #0 + ldur q2, [x10, #18] + sdot v20.4s, v16.16b, v17.16b + ldur q3, [x11, #18] + sdot v4.4s, v2.16b, v3.16b + ldur q18, [x10, #52] + ldur q19, [x11, #52] + ldr h21, [x10, #34] + sdot v20.4s, v18.16b, v19.16b + ldr h22, [x11, #34] + ldr h5, [x10], #68 + ldr h6, [x11], #68 + + fcvt s5, h5 + fcvt s6, h6 + addv s4, v4.4s + scvtf s4, s4 + fmul s5, s5, s6 + fmul s4, s4, s5 + + addv s20, v20.4s + fcvt s21, h21 + fcvt s22, h22 + scvtf s20, s20 + fmul s21, s21, s22 + fmul s20, s20, s21 + fadd s31, s31, s4 + fadd s31, s31, s20 + + sub w2, w2, #2 + cmp w2, #2 + b.ge .Lpairs + +.Ltail: + cbz w2, .Ldone + ldur q0, [x10, #2] + ldur q1, [x11, #2] + movi v4.4s, #0 + ldur q2, [x10, #18] + sdot v4.4s, v0.16b, v1.16b + ldur q3, [x11, #18] + ldr h5, [x10] + sdot v4.4s, v2.16b, v3.16b + ldr h6, [x11] + addv s4, v4.4s + fcvt s5, h5 + fcvt s6, h6 + scvtf s4, s4 + fmul s5, s5, s6 + fmul s4, s4, s5 + fadd s31, s31, s4 + +.Ldone: + str s31, [x9] + ret + +.Lbad_args: + mov w1, w7 + b vt_cpu_q8_dot_a76_bad_args + .cfi_endproc + .size vt_cpu_q8_dot_a76_asm, .-vt_cpu_q8_dot_a76_asm + .section .note.GNU-stack,"",%progbits diff --git a/src/vt/cpu/cpu_quant_dot_sdot.cpp b/src/vt/cpu/cpu_quant_dot_sdot.cpp new file mode 100644 index 000000000..799e8cae5 --- /dev/null +++ b/src/vt/cpu/cpu_quant_dot_sdot.cpp @@ -0,0 +1,156 @@ +// Cortex-A76 Q8_0 x Q8_0 DotProd tier (KERNEL-CPU-A76-Q8-DOT). +// +// The integer core follows llama.cpp @ 237ad9b96 +// ggml/src/ggml-cpu/arch/arm/quants.c:1076-1160, but preserves this project's +// stricter scalar per-block float accumulation order. The compiler arm exists +// to expose what GCC can do from ACLE; the assembly arm is a separately +// measurable AAPCS64 schedule over the exact same arithmetic. +#include "vt/quant.h" + +#if defined(VT_CPU_A76_Q8_DOT) && defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD) + +#include + +#if defined(__linux__) +#include +#include +#endif + +#include +#include +#include +#include + +#include "cpu_quant_blocks.h" + +#if defined(__linux__) && !defined(HWCAP_ASIMDDP) +#define HWCAP_ASIMDDP (1 << 20) +#endif + +extern "C" void vt_cpu_q8_dot_a76_asm(int n, float* s, size_t bs, const void* x, size_t bx, + const void* y, size_t by, int nrc); + +namespace vt::cpu { +namespace { + +bool CpuHasDotProd() { +#if defined(__linux__) + return (getauxval(AT_HWCAP) & HWCAP_ASIMDDP) != 0; +#else + return true; +#endif +} + +bool CpuIsCortexA76() { +#if defined(__linux__) + std::ifstream in("/sys/devices/system/cpu/cpu0/regs/identification/midr_el1"); + std::string value; + if (!(in >> value)) return false; + char* end = nullptr; + const unsigned long long midr = std::strtoull(value.c_str(), &end, 0); + if (end == value.c_str() || *end != '\0') return false; + const unsigned implementer = static_cast((midr >> 24) & 0xffU); + const unsigned part = static_cast((midr >> 4) & 0xfffU); + return implementer == 0x41U && part == 0xd0bU; +#else + return false; +#endif +} + +inline float HalfToFloat(uint16_t bits) { + _Float16 value; + static_assert(sizeof(value) == sizeof(bits)); + std::memcpy(&value, &bits, sizeof(value)); + return static_cast(value); +} + +float Q8DotSdot(const BlockQ8_0* x, const BlockQ8_0* y, int nb) { + float sumf = 0.0F; + for (int ib = 0; ib < nb; ++ib) { + int32x4_t dot = vdupq_n_s32(0); + dot = vdotq_s32(dot, vld1q_s8(x[ib].qs), vld1q_s8(y[ib].qs)); + dot = vdotq_s32(dot, vld1q_s8(x[ib].qs + 16), vld1q_s8(y[ib].qs + 16)); + const int sumi = vaddvq_s32(dot); + const float scale = HalfToFloat(x[ib].d) * HalfToFloat(y[ib].d); + sumf += static_cast(sumi) * scale; + } + return sumf; +} + +void CheckArgs(int n, int nrc, const char* name) { + VT_CHECK(n % kQK8_0 == 0, std::string(name) + ": n must be a multiple of 32"); + VT_CHECK(nrc == 1, std::string(name) + ": supports nrc == 1 only"); +} + +void VecDotQ8Sdot(int n, float* s, size_t bs, const void* vx, size_t bx, const void* vy, size_t by, + int nrc) { + CheckArgs(n, nrc, "vec_dot_q8_0_sdot"); + (void)bs; + (void)bx; + (void)by; + *s = Q8DotSdot(static_cast(vx), static_cast(vy), n / kQK8_0); +} + +} // namespace + +extern "C" [[noreturn]] void vt_cpu_q8_dot_a76_bad_args(int n, int nrc) { + CheckArgs(n, nrc, "vec_dot_q8_0_a76_asm"); + std::abort(); +} + +VecDotFn QuantQ8SdotVecDot() { + return CpuHasDotProd() ? &VecDotQ8Sdot : nullptr; +} + +VecDotFn QuantQ8A76AsmVecDot() { + return CpuHasDotProd() ? &vt_cpu_q8_dot_a76_asm : nullptr; +} + +VecDotFn SelectQuantQ8VecDot(VecDotFn portable) { + const char* value = std::getenv("VT_CPU_Q8_DOT"); + if (value == nullptr || std::strcmp(value, "auto") == 0) { + return QuantQ8A76AsmActive() ? QuantQ8A76AsmVecDot() : portable; + } + if (std::strcmp(value, "portable") == 0) { + return portable; + } + if (std::strcmp(value, "sdot") == 0) { + return QuantQ8SdotVecDot() != nullptr ? QuantQ8SdotVecDot() : portable; + } + if (std::strcmp(value, "a76-asm") == 0) { + return QuantQ8A76AsmVecDot() != nullptr ? QuantQ8A76AsmVecDot() : portable; + } + VT_CHECK(false, "VT_CPU_Q8_DOT must be auto, portable, sdot, or a76-asm"); + return portable; +} + +bool QuantQ8SdotActive() { + return QuantQ8SdotVecDot() != nullptr; +} +bool QuantQ8A76AsmActive() { + return QuantQ8A76AsmVecDot() != nullptr && CpuIsCortexA76(); +} + +} // namespace vt::cpu + +#else + +namespace vt::cpu { +VecDotFn QuantQ8SdotVecDot() { + return nullptr; +} +VecDotFn QuantQ8A76AsmVecDot() { + return nullptr; +} +VecDotFn SelectQuantQ8VecDot(VecDotFn portable) { + return portable; +} +bool QuantQ8SdotActive() { + return false; +} +bool QuantQ8A76AsmActive() { + return false; +} +} // namespace vt::cpu + +#endif diff --git a/tests/vt/test_ops_quant_dot.cpp b/tests/vt/test_ops_quant_dot.cpp index c92c866ef..fdc4f7354 100644 --- a/tests/vt/test_ops_quant_dot.cpp +++ b/tests/vt/test_ops_quant_dot.cpp @@ -204,6 +204,13 @@ float RunVecDot(vt::DType wtype, const uint8_t* wq, const uint8_t* aq, return s; } +float RunVecDotFn(vt::cpu::VecDotFn fn, const uint8_t* wq, + const uint8_t* aq, int64_t k) { + float s = 0.0F; + fn(static_cast(k), &s, 0, wq, 0, aq, 0, 1); + return s; +} + } // namespace // --------------------------------------------------------------------------- @@ -583,6 +590,73 @@ TEST_CASE("G3 dot_product_error within upstream bound (test-quantize-fns:86)") { CHECK(err < kMaxDotProductError); } +TEST_CASE("KERNEL-CPU-A76-Q8-DOT explicit SDOT and assembly match portable") { + const vt::cpu::VecDotFn sdot = vt::cpu::QuantQ8SdotVecDot(); + const vt::cpu::VecDotFn assembly = vt::cpu::QuantQ8A76AsmVecDot(); + CHECK((sdot == nullptr) == (assembly == nullptr)); + if (sdot == nullptr) { + CHECK_FALSE(vt::cpu::QuantQ8SdotActive()); + CHECK_FALSE(vt::cpu::QuantQ8A76AsmActive()); + return; + } + CHECK(vt::cpu::QuantQ8SdotActive()); + + const vt::cpu::VecDotFn portable = + vt::cpu::QuantTraits(vt::DType::kQ8_0).vec_dot; + for (int blocks : {1, 2, 3, 5, 64}) { + CAPTURE(blocks); + const int64_t k = 32 * blocks; + std::vector wq = + RandomBlocks(kWeightCases[1], blocks, 0xA760U + blocks); + std::vector act(static_cast(k)); + GenerateData(0.75F, act.size(), act.data()); + std::vector aq = + QuantizeActivation(vt::DType::kQ8_0, act.data(), k); + + // Offset both buffers by one byte: Q8 blocks are 34 bytes and therefore + // alternate natural alignment in real rows. All three tiers must accept + // an unaligned block base without reading beyond the final block. + std::vector wu(wq.size() + 2, 0xA5); + std::vector au(aq.size() + 2, 0x5A); + std::memcpy(wu.data() + 1, wq.data(), wq.size()); + std::memcpy(au.data() + 1, aq.data(), aq.size()); + const float ref = RunVecDotFn(portable, wu.data() + 1, au.data() + 1, k); + CHECK(RunVecDotFn(sdot, wu.data() + 1, au.data() + 1, k) == ref); + CHECK(RunVecDotFn(assembly, wu.data() + 1, au.data() + 1, k) == ref); + } + + auto check_edge_blocks = [&](uint16_t wd, uint16_t ad, bool zero_payload) { + constexpr int blocks = 2; + constexpr int block_bytes = 34; + std::vector wq(blocks * block_bytes); + std::vector aq(blocks * block_bytes); + for (int ib = 0; ib < blocks; ++ib) { + uint8_t* wb = wq.data() + ib * block_bytes; + uint8_t* ab = aq.data() + ib * block_bytes; + std::memcpy(wb, &wd, sizeof(wd)); + std::memcpy(ab, &ad, sizeof(ad)); + for (int j = 0; j < 32; ++j) { + wb[2 + j] = zero_payload + ? 0 + : static_cast((j & 1) != 0 ? 127 : -128); + ab[2 + j] = zero_payload + ? 0 + : static_cast((j & 2) != 0 ? -128 : 127); + } + } + const float ref = RunVecDotFn(portable, wq.data(), aq.data(), 64); + CHECK(RunVecDotFn(sdot, wq.data(), aq.data(), 64) == ref); + CHECK(RunVecDotFn(assembly, wq.data(), aq.data(), 64) == ref); + }; + check_edge_blocks(vt::F32ToF16(1.0F), vt::F32ToF16(1.0F), true); + check_edge_blocks(/*maximum finite f16=*/0x7BFFU, + /*minimum normal negative f16=*/0x8400U, false); + + std::vector one(34, 0); + float out = 0.0F; + CHECK_THROWS(sdot(33, &out, 0, one.data(), 0, one.data(), 0, 1)); + CHECK_THROWS(assembly(32, &out, 0, one.data(), 0, one.data(), 0, 2)); +} // --------------------------------------------------------------------------- // G3 — the GEMM wiring (kMatmulBTQuant), ported MUL_MAT cases // --------------------------------------------------------------------------- From 966b101ebb3ecdcbe8e34ab799f5e62f33040054 Mon Sep 17 00:00:00 2001 From: Richard Palethorpe Date: Thu, 6 Aug 2026 23:26:35 +0100 Subject: [PATCH 7/7] Benchmark Cortex-A76 CPU against llama.cpp Record the same-file Raspberry Pi 5 speed, memory, and exact-output comparison, including the reproducible b9892 reconstruction and void intrusive-sampler attempt. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 2 +- .agents/backend-matrix.md | 4 +- .agents/feature-matrix.md | 2 +- .agents/kernel-matrix.md | 2 +- .agents/parity-ledger.md | 1 + .agents/roadmap_v1.md | 2 +- .agents/specs/cpu-a76-q8-dot.md | 15 ++- .../specs/rpi5-cortex-a76-cpu-optimization.md | 28 ++++- .agents/state.md | 28 +++++ docs/BENCHMARKS.md | 16 ++- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- .../rpi5-a76-llamacpp-20260806.md | 117 ++++++++++++++++++ .../rpi5-a76-q8-dot-20260806.md | 12 +- 14 files changed, 210 insertions(+), 23 deletions(-) create mode 100644 docs/bench-evidence/rpi5-a76-llamacpp-20260806.md diff --git a/.agents/NOW.md b/.agents/NOW.md index 68578bfe4..7bb63d87e 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| RPi5 A76 CPU | **R5 ASSEMBLY GREEN:** output-exact AAPCS64 beats compiler SDOT 3.66-5.08%; Qwen TTFT/E2E improve | W6: M1/T4 −2.43%, BF16 GEMM, memory/concurrency and llama.cpp floor | +| RPi5 A76 CPU | **R5 ASSEMBLY GREEN; llama floor MEASURED/NOT MET:** exact AAPCS64 beats SDOT, but vllm.cpp is 0.461x prefill / 0.653x decode+E2E vs llama.cpp; RSS 24.2% lower | W6: profile the 2.17x prefill / 1.53x decode gap, starting with BF16 GEMM; M1/T4 and concurrency remain | | MXFP4 parity | **TERMINAL:** c1 1.020x pass; c2-c8 0.962-0.969 GPU-intrinsic; final cap/glue levers exhausted | Record; branch unmerged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 08d786c3a..ac87aabef 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -223,7 +223,7 @@ on every listed target. | `BACKEND-ABI-VT` | Backend registry, device/queue, capture, drop-in adapter ABI | platform contract `vllm/platforms/interface.py:67-229`; stable stream `csrc/libtorch_stable/torch_utils.h:76-82`; raw Marlin boundary `csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu:344-356` | queue ID [device.h:9](../include/vt/device.h#L9), explicit resources [backend.h:53](../include/vt/backend.h#L53), scalar/layout ABI [ops.h:13](../include/vt/ops.h#L13), CUDA workspace/raw probe [cuda_dropin.cu:104](../src/vt/cuda/cuda_dropin.cu#L104) | [ported ABI tests](../tests/vt/test_dropin_abi.cpp#L86); CPU 94/94 + repair-focused 1/1; GCC13 repair at [test_dropin_abi.cpp:245](../tests/vt/test_dropin_abi.cpp#L245); exact sm_121a all-target build 100%, focused CUDA/ABI CTest 2/2, sanitizer 9/9 + 196/196 with 0 errors/leaks, 35B/27B gates 2/2 (`1141b79`, evidence manifest `4adbe952…601`) | [drop-in ABI](specs/dropin-kernel-abi.md) | `ANCHOR-BACKFILL` - sm_80/sm_90a cross-build and unchanged-trace/model A/B-memory proof remain; common scalar forwarder and legacy backend-shim migration remain; no production family migrated | CLAIM-BACKEND-ABI-W0-GPU-1 | | `BACKEND-ATTN-REGISTRY` | Attention-backend registry + platform-driven priority selection (extensibility **item 4** — the second portability seam, now REALIZED). Backends SELF-REGISTER per `(DeviceType, name)`; `Platform::get_attn_backend_priority()` advertises a capability-ordered name list; the selector returns the first REGISTERED name. Selection is DATA (register + priority slot), not an inline code edit. The concrete attention KERNEL stays at the vt:: op-table (`vt::PagedAttention`→`GetOp`), already device-additive | `vllm/v1/attention/backends/registry.py` (self-registration), `vllm/platforms/cuda.py:361-470` (`get_valid_backends`/`get_attn_backend_cls`) + `:84-166` (`_get_backend_priorities`, non-MLA major-10 vs else), `cpu.py:75-87` (CPU_ATTN) @ pin `e24d1b24` | registry+selector [registry.h:44](../include/vllm/v1/attention/registry.h#L44), [registry.cpp:60](../src/vllm/v1/attention/registry.cpp#L60); priority [interface.h:92](../include/vllm/platforms/interface.h#L92), CUDA major-10/else [cuda.cpp:44](../src/vllm/platforms/cuda.cpp#L44), CPU [cpu.cpp:29](../src/vllm/platforms/cpu.cpp#L29); FLASH_ATTN self-reg [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), GDN [gdn_attn.cpp:153](../src/vllm/v1/attention/backends/gdn_attn.cpp#L153) | [registry tests](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L61) (self-register + Make/throw + CUDA/CPU priority order + first-registered walk + override; 8 cases/25 assertions) + [platform priority](../tests/vllm/platforms/test_platform.cpp#L53); clean CPU `-Werror` + full CPU CTest green; behavior-preserving ⇒ same FA2 attention selected (**DGX-CONFIRMED @ `2c732e7`: 27B 235/235 + 35B 315/315 token-exact, FA2 sm_121a, memcheck 0/315**) | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) (item 4) | `ANCHOR-BACKFILL` — item-4 seam LANDED CPU + DGX-CONFIRMED; FLASHINFER/TRITON_ATTN/FLEX/TURBOQUANT are named in the priority (fidelity) but unimplemented ⇒ walk falls through to FLASH_ATTN (the behavior-preserving outcome); MLA-branch priorities deferred until an MLA model ports | CLAIM-ATTN-REGISTRY-1 | | `BACKEND-PLATFORM` | Platform capability/memory-model seam (`is_cuda`/`is_cpu`, `is_unified_memory`, `has_device_capability`, `supported_dtypes`, `residency_policy`, `supports_graph_capture`; `get_attn_backend_priority` REALIZED by `BACKEND-ATTN-REGISTRY` item 4; **+ the S3 capability predicates `supports_fp8`/`cutlass_fp4_supported`/`opaque_attention_op`/`is_integrated_gpu`/`support_static_graph_mode`/`is_device_capability_family`, mirroring `interface.py:914,933,977,1058,441-476` + `nvfp4_utils.py:56`, added by `CLAIM-BACKEND-SEAM-S3-1` and consumed by the fp4/fp8/graph gates in `qwen3_5.cpp`**); self-registered per `DeviceType` via `CurrentPlatform()` | `vllm/platforms/interface.py:134-229` (`class Platform`), `:409-439` (get/has_device_capability), `:181-187` (supported_dtypes) @ pin `e24d1b24` | interface [interface.h:56](../include/vllm/platforms/interface.h#L56), registry/CurrentPlatform [platform.cpp:57](../src/vllm/platforms/platform.cpp#L57), CPU [cpu.cpp:11](../src/vllm/platforms/cpu.cpp#L11), CUDA [cuda.cpp:14](../src/vllm/platforms/cuda.cpp#L14); composes `vt::Backend` [backend.h:22](../include/vt/backend.h#L22) | [platform tests](../tests/vllm/platforms/test_platform.cpp#L28) (registration + CPU capabilities + has_device_capability), clean CPU `-Werror` + full CPU CTest; 7 memory-model/residency sites migrated (`runner.cpp`, `model_registry.cpp`, `qwen3_5.cpp`) — **per-tensor sites key on `GetPlatform(.device.type).is_cuda()` (the OBJECT's device), NOT the process-global `CurrentPlatform()`, which is accelerator-first and would mis-route a CPU queue/tensor on a GPU box (regression fixed 2026-07-18; red DGX CPU tests `test_platform`/`test_qwen27_dense_forward`); `CurrentPlatform()` is reserved for process-level "which accelerator" questions only**. DGX behavior-preserving model gates 27B 235/235 + 35B 315/315 CONFIRMED @ (CUDA -Werror-clean after a GCC13 dangling-pointer FP fix). **Item-2 residency_policy() CONSUMPTION LANDED (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`):** the host-free / load-stream / DevicePool-cap decisions in `qwen3_5.cpp` now READ `GetPlatform(.device.type).residency_policy()` (per-device) instead of an inline `device.type`/env gate — host-free `qwen3_5.cpp:BuildMoeMarlinResident` via `ShouldReleaseHostWeights`, load-stream `Qwen3_5Model::PrepareMarlinResident` via `ShouldInterleaveLoadStream` (both `interface.h`), DevicePool soft cap via `residency_policy().device_pool_cap_bytes` (`DBuf`/`DevicePool`). `CudaPlatform::residency_policy().release_host_weights_after_upload` flipped **false→true** (now CONSUMED ⇒ reproduces today's GB10 host-free-after-Marlin-build EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH gate; `VT_MOE_HOST_FREE`/`VT_MOE_LOADSTREAM` env stay as overrides. Consumption unit-tested `test_platform.cpp` (7 cases/43 assertions). A new (discrete) GPU sets `residency_policy()` values ⇒ ZERO model edit. **DGX gate PASSED @ `62fc0e0` (production flags CUTLASS sm120a + FA2 sm_121a + Triton AOT, one flock): clean CUDA `-Werror` 0 warn, 27B 235/235 + 35B 315/315 token-exact, 35B VmHWM ≈ 4.0 GiB (load-stream win preserved), memcheck 0 errors.** | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md) | `ANCHOR-BACKFILL` — item-1 extraction DONE + DGX-CONFIRMED; item-2 residency consumption LANDED + DGX-PROVEN (`CLAIM-BACKEND-PLATFORM-2`); kernel-shape dispatch branches deliberately left for items 4/5 | CLAIM-BACKEND-PLATFORM-1 / CLAIM-BACKEND-PLATFORM-2 | -| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06:** R0-R3 pin the exact Q8_K_XL vehicle, local buildx/QEMU build and physical-Pi correctness/PMU baseline. **R4-R5 GREEN:** [KERNEL-CPU-A76-Q8-DOT](kernel-matrix.md) adds exact-order compiler SDOT and a scheduled AAPCS64 leaf; Pi operation and 64-token model outputs are exact. Assembly beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4, lowers Qwen TTFT 1.55% and E2E 0.13%, so A76+DotProd auto-selects it. M1/T4 regresses 2.43%; thread partition, peak memory, concurrency and same-file Pi llama.cpp floor stay open. No 20-core/i8mm result transfers. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md) | `PARTIAL` | PR #65 | +| `BACKEND-CPU` | CPU correctness and production path | `platforms/cpu.py:42-125`, CPU ops rooted at `csrc/cpu/torch_bindings.cpp:123-139`; llama.cpp `ggml-cpu.c:471-610,3024-3390` owns the added native pool reference | [cpu_backend.cpp:11](../src/vt/cpu/cpu_backend.cpp#L11), [threadpool.cpp:78](../src/vt/cpu/cpu_threadpool.cpp#L78), [chunked ops](../src/vt/cpu/cpu_ops.cpp#L88) | [backend tests](../tests/vt/test_backend.cpp#L10), [op parity](../tests/parity/test_op_parity.cpp#L34), [threadpool/upstream-test port](../tests/vt/test_cpu_threadpool.cpp#L63); 1/3/20 full CPU suites + TSAN pass. The binding 20-core Arm/i8mm Qwen3.5-2B Q8 single-stream gate is at or ahead of same-file llama.cpp on prefill/decode/RSS; server concurrency remains open. **Raspberry Pi 5 / Cortex-A76 lane GATING 2026-08-06:** R0-R3 pin the exact Q8_K_XL vehicle, local buildx/QEMU build and physical-Pi correctness/PMU baseline. **R4-R5 GREEN:** [KERNEL-CPU-A76-Q8-DOT](kernel-matrix.md) adds exact-order compiler SDOT and a scheduled AAPCS64 leaf; Pi operation and 64-token model outputs are exact. Assembly beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4, lowers Qwen TTFT 1.55% and E2E 0.13%, so A76+DotProd auto-selects it. M1/T4 regresses 2.43%. **The separate Pi llama.cpp floor is now MEASURED/NOT MET on speed:** vllm.cpp is 0.461x prefill / 0.653x decode+E2E, while using 24.2% less RSS; exact-prompt 64-token output matches. No 20-core/i8mm result transfers. Thread partition, BF16 GEMM/speed closure and concurrency stay open. **macOS UNBLOCKED 2026-07-22 (`CLAIM-BACKEND-FANOUT-1`, W0):** the CPU registrar force-link is fixed; `test_backend` is **7/7 (18/18)** and the whole tree builds `-Werror`-clean on M4. Two unrelated macOS test gaps remain recorded in [environment.md](environment.md) | [threadpool leaf](specs/gguf-cpu-threadpool.md); [RPi5 Cortex-A76 campaign](specs/rpi5-cortex-a76-cpu-optimization.md); [Pi competitor evidence](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md) | `PARTIAL` | PR #65 | | `BACKEND-CPU-ZEN` | AMD Zen CPU with ZenDNN/zentorch dispatch and weight prepack | `platforms/zen_cpu.py:12-32`; detection `platforms/__init__.py:153-192`; `tests/test_zen_cpu_platform_detection.py:8-37` | generic CPU backend only; no Zen-specific dispatch | - | [CUDA inventory](specs/cuda-architecture-inventory.md); leaf spike required | `INVENTORIED` | - | | `BACKEND-ROCM` | AMD ROCm/HIP | `platforms/__init__.py:110-128`, `platforms/rocm.py:43-125`, ROCm ops rooted at `csrc/rocm/moe_q_gemm_rdna3.cu:1` | Lands through the realized seams: a `platforms/rocm.cpp` Platform + a `kROCM` attention TU self-registering via the [attn-registry seam](../include/vllm/v1/attention/registry.h#L44) + a `rocm.cpp` `get_attn_backend_priority()` slot (mirror `rocm.py`'s ROCM_ATTN/AITER ordering); ZERO selector/model/runner edit | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-XPU` | Intel XPU loyal port; gating exploration E4 ([backends.md](backends.md) kernel sourcing: SYCL-vs-Level-Zero call + whether upstream XPU attention contracts translate 1:1; explore when T2 scheduling begins) | `platforms/__init__.py:131-150`, `platforms/xpu.py:103-125` | enum slot only [device.h:11](../include/vt/device.h#L11) | [unavailable-backend test](../tests/vt/test_backend.cpp#L37); **SPIKE-ONLY verdict, MEASURED 2026-07-22: HW-BLOCKED — no Intel GPU on any box we own, and no acquisition path recorded.** Compounding finding: the loyalty target is INCOMPLETE upstream — `vllm/platforms/xpu.py:11-13` imports the EXTERNAL `vllm_xpu_kernels._C`/`._moe_C`/`._xpu_C`, and a search of the pinned tree for `*.sycl`/`*.dp.cpp` returns NOTHING, so there is no in-tree SYCL kernel source to mirror 1:1. What stays doable with zero Intel hardware: the attention-selector/priority POLICY port (pure data, unit-testable), oneAPI DPC++ compile coverage, and SYCL kernels executed on the oneAPI OpenCL CPU device for unit-level numerics. **No e2e or performance gate is proposed, because none can be run.** No XPU backend code exists | [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `SPIKE` | `CLAIM-BACKEND-FANOUT-1` | @@ -248,7 +248,7 @@ memory. Floating competitor versions do not count. | `BACKEND-GATE-ROCM-VLLM` | ROCm parity vs vLLM | pinned vLLM ROCm backend | - | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-GATE-ROCM-SGLANG` | ROCm low-concurrency serving vs SGLang | pinned SGLang ROCm build | - | - | [competitive benchmark spike](specs/competitive-benchmarks.md) | `INVENTORIED` | - | | `BACKEND-GATE-XPU-VLLM` | Intel XPU parity vs vLLM | pinned vLLM XPU backend | - | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | -| `BACKEND-GATE-CPU-LLAMACPP` | CPU throughput/latency/memory vs llama.cpp | [llama.cpp](https://github.com/ggml-org/llama.cpp), commit pinned per run | [CPU backend:11](../src/vt/cpu/cpu_backend.cpp#L11), [bench main:1](../examples/bench/main.cpp#L1) | [CPU correctness](../tests/parity/test_op_parity.cpp#L34); **RE-MEASURED 2026-07-22** on Qwen3.5-2B Q8 GGUF, same file, BINDING arm = idle `dgx.casa` aarch64 under one `flock`: llama.cpp ahead **11.6× decode / 33.5× prefill / 2.65× peak RSS** (7.427 vs 2.798 GiB), reps within 0.36–0.59 % ([floor re-measurement](specs/cpu-llamacpp-floor-remeasure-2026-07-22.md)). SUPERSEDES the B4 54–75× / ≈1,480× figures ([ledger B4 row](parity-ledger.md#L290)), which predated the threadpool; the x86 1-thread arm reproduces B4 within 4.8–5.5 %, so the movement is entirely W1–W3. Peak RSS UNMOVED. Gap ATTRIBUTED: `kMatmul` was **95.4 %** of wall time. **TWO LEVERS HAVE SINCE LANDED** on that attribution, both binding on the same host/file/recipe: `QUANT-GGUF-CIQ-GEMM` **G4** (route block weights to the quant GEMM) took it to 3.38× / 8.20× / 2.29×, and `KERNEL-GEMM-CPU-ELEM` **E1-E4** ([elementwise CPU GEMM](specs/cpu-elementwise-gemm.md), 2026-07-22) took it to **decode 1.03× (AT PARITY, within 3.1 % and inside llama.cpp's own run spread), prefill 2.34×, peak RSS 2.29×** — llama.cpp pp128 173.28±1.75, tg32 24.52±0.45, RSS 2.798 GiB vs ours 73.97 / 23.79 t/s / 6.401 GiB, tokens byte-identical throughout. **Gate stays OPEN on prefill and RSS.** The E1-E4 attribution is now STALE — M-blocking gave 1.63× op-level prefill and 0.0 % end-to-end. **FOUR MORE LEVERS HAVE SINCE LANDED against fresh profiles, all binding on the same host/file/recipe, all bit-identical (token md5 `d235db12…` unchanged):** keep-quant loader **L5** RSS 6.401→3.884 GiB (2.29×→**1.39×**); **thread kGdnPrefill + kPagedAttention** prefill 2.43×→1.76× behind; **CIQ G6** Arm i8mm quant tier 1.56×→1.44× behind; and **GDN split-projection orientation** (2026-07-23, `CLAIM-CPU-GDN-ORIENT-1`) — a fresh op-dispatch profile found the four GDN input projections (72 GEMMs, 17.9 % of prefill) were the last weight family transposed into the slow [K,N] `kMatmul`; keeping them [N,K] (nk=true) → M-blocked `kMatmulBT` is same-binary prefill **1.090×** and decode 1.09×, **1.44× → 1.32× behind** llama.cpp pp128, RSS unchanged, tokens byte-identical, `kMatmul` eliminated from prefill (72→0 calls). **TWO FINAL LEVERS THEN CLOSED THE GATE (2026-07-23):** **CIQ G7** (q8_0 repack-at-load) crossed prefill parity — same-binary 1.92× → 223.8 t/s = **1.26× AHEAD** of llama.cpp pp128; and **keep-quant loader L7** (`CLAIM-QUANT-GGUF-RSS-L7-1`) CLOSED the RSS gap — a per-pool profile disproved L6's "engine workspace" attribution (DevicePool 20 MiB, whole KV 115 MiB, both ≤ llama.cpp), found the residual was a q8_0 repack-source double-count on Arm (kept resident because keep-f16 holds the mmap open), and releasing the dead source (port of `unmap_fragment`) + prefaulting borrowed weights (port of the mmap prefetch) + flipping keep-f16 default-on took peak RSS **3.884 → 2.832 GiB = 1.39× → 1.01× (PARITY)**, prefill **1.18× AHEAD** (204 vs pp128 173.2), decode ~parity (24.4 vs 25.09), tokens byte-identical (md5 `809f2d0…`). **Current CPU position: at OR ahead of llama.cpp on EVERY axis — peak RSS 1.01× (parity), prefill 1.18–1.26× ahead, decode at parity — the CPU gaps are FULLY CLOSED** on single-request `Qwen3.5-2B` Q8 GGUF (server-endpoint + batched operating points remain future scope) | [competitive benchmark spike](specs/competitive-benchmarks.md); [floor re-measurement](specs/cpu-llamacpp-floor-remeasure-2026-07-22.md); [elementwise CPU GEMM](specs/cpu-elementwise-gemm.md); [GDN projection orientation](specs/cpu-gdn-proj-orientation-2026-07-23.md); [keep-quant loader L7](specs/gguf-keep-quant-loader.md) | `INVENTORIED` | - | +| `BACKEND-GATE-CPU-LLAMACPP` | CPU throughput/latency/memory vs llama.cpp | [llama.cpp](https://github.com/ggml-org/llama.cpp), commit pinned per run | [CPU backend:11](../src/vt/cpu/cpu_backend.cpp#L11), [bench main:1](../examples/bench/main.cpp#L1) | [CPU correctness](../tests/parity/test_op_parity.cpp#L34); **RE-MEASURED 2026-07-22** on Qwen3.5-2B Q8 GGUF, same file, BINDING arm = idle `dgx.casa` aarch64 under one `flock`: llama.cpp ahead **11.6× decode / 33.5× prefill / 2.65× peak RSS** (7.427 vs 2.798 GiB), reps within 0.36–0.59 % ([floor re-measurement](specs/cpu-llamacpp-floor-remeasure-2026-07-22.md)). SUPERSEDES the B4 54–75× / ≈1,480× figures ([ledger B4 row](parity-ledger.md#L290)), which predated the threadpool; the x86 1-thread arm reproduces B4 within 4.8–5.5 %, so the movement is entirely W1–W3. Peak RSS UNMOVED. Gap ATTRIBUTED: `kMatmul` was **95.4 %** of wall time. **TWO LEVERS HAVE SINCE LANDED** on that attribution, both binding on the same host/file/recipe: `QUANT-GGUF-CIQ-GEMM` **G4** (route block weights to the quant GEMM) took it to 3.38× / 8.20× / 2.29×, and `KERNEL-GEMM-CPU-ELEM` **E1-E4** ([elementwise CPU GEMM](specs/cpu-elementwise-gemm.md), 2026-07-22) took it to **decode 1.03× (AT PARITY, within 3.1 % and inside llama.cpp's own run spread), prefill 2.34×, peak RSS 2.29×** — llama.cpp pp128 173.28±1.75, tg32 24.52±0.45, RSS 2.798 GiB vs ours 73.97 / 23.79 t/s / 6.401 GiB, tokens byte-identical throughout. **Gate stays OPEN on prefill and RSS.** The E1-E4 attribution is now STALE — M-blocking gave 1.63× op-level prefill and 0.0 % end-to-end. **FOUR MORE LEVERS HAVE SINCE LANDED against fresh profiles, all binding on the same host/file/recipe, all bit-identical (token md5 `d235db12…` unchanged):** keep-quant loader **L5** RSS 6.401→3.884 GiB (2.29×→**1.39×**); **thread kGdnPrefill + kPagedAttention** prefill 2.43×→1.76× behind; **CIQ G6** Arm i8mm quant tier 1.56×→1.44× behind; and **GDN split-projection orientation** (2026-07-23, `CLAIM-CPU-GDN-ORIENT-1`) — a fresh op-dispatch profile found the four GDN input projections (72 GEMMs, 17.9 % of prefill) were the last weight family transposed into the slow [K,N] `kMatmul`; keeping them [N,K] (nk=true) → M-blocked `kMatmulBT` is same-binary prefill **1.090×** and decode 1.09×, **1.44× → 1.32× behind** llama.cpp pp128, RSS unchanged, tokens byte-identical, `kMatmul` eliminated from prefill (72→0 calls). **TWO FINAL LEVERS THEN CLOSED THE GATE (2026-07-23):** **CIQ G7** (q8_0 repack-at-load) crossed prefill parity — same-binary 1.92× → 223.8 t/s = **1.26× AHEAD** of llama.cpp pp128; and **keep-quant loader L7** (`CLAIM-QUANT-GGUF-RSS-L7-1`) CLOSED the RSS gap — a per-pool profile disproved L6's "engine workspace" attribution (DevicePool 20 MiB, whole KV 115 MiB, both ≤ llama.cpp), found the residual was a q8_0 repack-source double-count on Arm (kept resident because keep-f16 holds the mmap open), and releasing the dead source (port of `unmap_fragment`) + prefaulting borrowed weights (port of the mmap prefetch) + flipping keep-f16 default-on took peak RSS **3.884 → 2.832 GiB = 1.39× → 1.01× (PARITY)**, prefill **1.18× AHEAD** (204 vs pp128 173.2), decode ~parity (24.4 vs 25.09), tokens byte-identical (md5 `809f2d0…`). **Current 20-core/i8mm position: at OR ahead of llama.cpp on every axis — peak RSS 1.01x (parity), prefill 1.18-1.26x ahead, decode at parity — that host-specific gate is closed. The four-core Cortex-A76/RPi5 arm is separately measured and OPEN on speed:** vllm.cpp 0.461x prefill / 0.653x decode+E2E, but 0.758x peak RSS (better), with same-text 64-token greedy output byte-identical ([Pi evidence](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md)). Server-endpoint + batched operating points remain future scope | [competitive benchmark spike](specs/competitive-benchmarks.md); [floor re-measurement](specs/cpu-llamacpp-floor-remeasure-2026-07-22.md); [elementwise CPU GEMM](specs/cpu-elementwise-gemm.md); [GDN projection orientation](specs/cpu-gdn-proj-orientation-2026-07-23.md); [keep-quant loader L7](specs/gguf-keep-quant-loader.md) | `INVENTORIED` | - | | `BACKEND-GATE-CUDA-LLAMACPP-LEGACY` | **Beyond-vLLM breadth floor** — Pascal/Volta/Turing throughput/latency/memory vs **llama.cpp on the same old card** (T4/V100/P40/P100). vLLM has NO entry on these arches (it drops them), so llama.cpp is the ONLY competitor floor here; it is also our port source | [llama.cpp](https://github.com/ggml-org/llama.cpp) `237ad9b96` `ggml-cuda` (attention `fattn-tile`/`fattn-vec`, GEMM `mmvf`/cuBLAS), pinned per run | the fp16/non-tensor-core lane [cuda_paged_attn.cu:706](../src/vt/cuda/cuda_paged_attn.cu#L706) (blocker) + [:2349](../src/vt/cuda/cuda_paged_attn.cu#L2349) (existing scalar fallback) | no run — no Turing/Volta/Pascal card here, and no `<13` toolkit for Volta/Pascal; correctness reference is llama.cpp-on-card + a portable/newer-card cross-check (NO vLLM oracle runs on these cards) | [CUDA arch breadth fp16](specs/cuda-arch-breadth-fp16.md) | `INVENTORIED` | - | | `BACKEND-GATE-METAL-OMLX` | MLX/Metal serving vs oMLX | [oMLX v0.5.0rc1](https://github.com/jundot/omlx/releases/tag/v0.5.0rc1), then pinned successor | - | - | [competitive benchmark spike](specs/competitive-benchmarks.md) | `INVENTORIED` | - | | `BACKEND-GATE-METAL-MLXLM` | **MLX is the NAMED COMPETITOR FLOOR for Metal (USER DIRECTIVE 2026-07-22: "we should do benchmark testing against MLX for the same model").** Binds exactly as AGENTS.md Acceptance rule defines for vLLM on CUDA: SAME model, SAME workload, SAME box, ours >= MLX on EVERY axis (total + output throughput, req/s, TTFT, TPOT/ITL, peak memory), correctness a precondition never traded off | [MLX-LM](https://github.com/ml-explore/mlx-lm), commit pinned per run | - | **NOT YET RUNNABLE — nothing to benchmark: the Metal backend is a W0 SKELETON with no model** (`BACKEND-METAL-MLX`). No number is claimed or owed. **Do NOT conflate MLX's two roles:** as an IMPLEMENTATION path it stays DEMOTED to work row M5 (bring-up is native MSL, zero installs); as a BENCHMARK ARM it is binding from the first Metal model onward. **HARD CONSTRAINT ON M3:** the first Metal bring-up model must be one MLX can also run, or the arms are incomparable — this composes with the spike's "OPT or Qwen3-dense first, NEVER Qwen3.5-Next", and since MLX-LM ships Qwen3 dense support, **Qwen3-dense is the pairing satisfying both**. **M4 readiness probed 2026-07-22 (read-only, NOT installed):** `brew info mlx` = 0.32.0, not installed, sole dep `python@3.14`; system python `/usr/bin/python3` 3.9.6 (CLT) with `venv` present; `import mlx` fails; ~28 GiB free. Deliberately NOT installed during W0 because the brew formula puts `python@3.14` in `/opt/homebrew/bin`, first on the PATH our macOS builds use, which would change `find_package(Python3)` and perturb the very gates W0 was proving. **Recommended (venv, avoids that hazard):** `python3 -m venv ~/mlx-venv && ~/mlx-venv/bin/pip install -U pip mlx-lm`; record the exact resolved version — an unpinned competitor arm is not a floor. **Any M4 timing is VOID unless the `com.localai.worker` root LaunchDaemon is booted out first** (needs interactive sudo; commands in [environment.md](environment.md)). **MLX INSTALLED AND AN UNOPPOSED BASELINE MEASURED 2026-07-22** ([study §7](specs/metal-mlx-reuse-study.md), [BENCHMARKS](../docs/BENCHMARKS.md)) — venv route as recommended, brew NOT used, so `python@3.14` never touched the build PATH. Resolved **`mlx` 0.29.3 / `mlx-metal` 0.29.3 / `mlx-lm` 0.29.1** (CLT python 3.9.6 caps the resolve below brew's 0.32.0). Model **`mlx-community/Qwen3-1.7B-bf16` @ `9cd6692855d3e06772228e9a962b2606359b2d24`** — 1.7B not 4B/8B because the desktop session already holds ~13 of 17.2 GB and a b=16 arm must not page. Harness is **MLX-LM's OWN** `python -m mlx_lm.benchmark`, p=512 g=128, 1 warmup + 3 timed trials. **MLX numbers (aggregate over batch):** b=1 27.57 tok/s decode / 470 ms TTFT / 36.3 ms ITL / 3.78 GB peak; b=2 48.91 / 900 ms / 40.9 ms / 3.97 GB; b=4 90.15 / 1738 ms / 44.4 ms / 4.18 GB; b=8 156.95 / 3416 ms / 51.0 ms / 4.47 GB; b=16 **213.39** / 6857 ms / 75.0 ms / 5.28 GB. Prefill saturates ~1200 tok/s by b=8 (compute roof); decode scales **7.74x** from b=1 to b=16 (bandwidth-bound, weight traffic amortized). Trial spread **0.12%-0.63%** across all five arms. **STATUS = `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** the worker could not be stopped (`sudo -n true` -> "a password is required"), so per the standing contended-run rule these may NOT be cited as a bound floor — they are a DESIGN TARGET plus an exact re-run recipe. The worker was nevertheless MEASURED genuinely idle (0.0% CPU; `ioreg` Device/Renderer/Tiler Utilization all **0**), consistent with the sub-1% spread; a second unanticipated contender WAS found — the desktop aerial video wallpaper (`WallpaperAerialsExtension` 8.2% CPU + `VTDecoderXPCService`), which must also be disabled for a binding run. **NO "ours" column exists or was manufactured — no Metal model runs, and no Metal speed result is claimed. THE ROW STAYS `INVENTORIED` DELIBERATELY:** a competitor baseline is not a gate. There is no implementation code and no ours-vs-MLX evidence to anchor, so promoting the state would be exactly the ungrounded transition `scripts/check-agent-record.py` exists to reject — the MLX arm is now READY and PINNED, and the row moves only when a Metal model runs (work row `M3b`). **=== OURS-VS-MLX PRODUCED 2026-07-23 (M3b) — INDICATIVE / `BLOCKED-ON-SUDO`, and the row moves `INVENTORIED` -> `ACTIVE`. ===** Qwen3-dense now runs on Metal, so both arms exist for the SAME model. Both = Qwen3-1.7B bf16 (ours = `Qwen/Qwen3-1.7B` HF safetensors on our Metal engine, device=2 confirmed via `VT_OP_PROVIDER_STATS` — every op `vt-native`; MLX = `mlx-community/Qwen3-1.7B-bf16`), SAME box, SAME session, p=512 g=128, b∈{1,2,4,8,16}. **Ours (`vllm-bench`, Metal), decode tok/s per stream · TTFT · peak:** b=1 4.29 · 4.86 s · 7.36 GB; b=2 4.32 · 9.08 s · 7.36; b=4 4.16 · 17.66 s · 7.37; b=8 3.25 · 28.52 s · 7.64; b=16 2.14 · 47.17 s · 8.78. Aggregate output tok/s 3.71 -> 19.14. **MLX (re-run same session, matches the committed §7 baseline):** gen 27.77 -> 211.55 tok/s, TTFT 0.47 -> 7.16 s, peak 3.78 -> 5.28 GB. **VERDICT: ours LOSES on every axis — ~6-11x slower decode, ~7-10x slower TTFT, ~2x peak memory.** This is a knowingly-unoptimised FLOOR, NOT our best: one command buffer per op (commit+wait), a plain threadgroup-tiled GEMM with no simdgroup-matrix, no batched encoders (`M3c`) — vs MLX's `steel` kernels. It is the FIRST ours-vs-MLX number and it sets the optimisation target. **STILL INDICATIVE, NOT BINDING:** the Mac could NOT be quieted (`sudo -n true` -> password required; the root `com.localai.worker` daemon and the aerial wallpaper `WallpaperAerialsExtension` ~9.8% CPU are both up), so per the standing contended-run rule the numbers are void as a bound comparison — though memory pressure read 83% free and MLX's trial spread stayed sub-1%. **The row is `ACTIVE` (not `DONE`): a native-competitor gate is `DONE` only when a model runs token-exact AND at MLX-parity-or-better on a genuinely quiet box; ours meets neither the parity nor the quiet-box condition. Exact residual: close the ~6-11x decode / ~7-10x TTFT / ~2x memory gap (start with `M3c` + a simdgroup GEMM) AND re-run once the user boots out the daemon + wallpaper (commands in [environment.md](environment.md)).** **=== MLX-LM FLOOR RE-MEASURED 2026-07-27, AND IT REPRODUCES. ===** Same box, same model revision, byte-identical toolchain (`mlx` 0.29.3 / `mlx-metal` 0.29.3 / `mlx-lm` 0.29.1), MLX-LM's own harness, p=512 g=128 n=3, this time with the three `actions.runner` LaunchAgents verified job-idle, booted out and restored, and the sweep under the GPU lock. **gen tok/s: b=1 27.89, b=2 49.69, b=4 91.10, b=8 158.65, b=16 213.71**; prompt tok/s 1091.2 / 1139.5 / 1180.2 / 1200.2 / 1197.7; peak memory 3.776 / 3.974 / 4.184 / 4.466 / 5.279 GB. Against the committed 2026-07-22 baseline this is **+0.15% to +1.6% on generation, within 0.3% on prefill, and peak memory IDENTICAL TO THREE DECIMALS at every batch size** — the ~1% is consistent with the paused runners. Trial spread tightened to **0.05%-0.28%** (was 0.12%-0.63%). **The competitor floor is therefore REPRODUCED, not merely restated.** **OURS, SAME SESSION:** MLX-provider arm 5.79 / 10.19 / 15.75 / 25.70 / 38.65 aggregate tok/s; native MSL arm 3.08 / 6.08 / 10.24 / 13.69 / 17.69. **Gap vs MLX-LM: 4.8x-6.2x with the MLX GEMM provider, 8.2x-12.1x without.** TTFT 2.7x-7.1x behind, peak memory 1.4x-1.8x higher. **WHY THE GAP PERSISTS IS NOW ANSWERED BY MEASUREMENT, NOT NARRATIVE** ([Metal dispatch attribution](specs/metal-dispatch-attribution.md)): the per-op `commit`+`waitUntilCompleted` model costs ~186 us x ~395 dispatches per decode token, a **~13.6 tok/s hard ceiling with infinitely fast kernels** versus MLX-LM's measured 27.9. **This row cannot pass on kernel work alone; `M3c` (batched encoders) is the precondition.** MLX commits at the `eval()` boundary rather than the op boundary, paying the same round trip a handful of times per token where we pay it ~395 times. Row stays `ACTIVE`: still short on every axis, and still not a quiet box. | [competitive benchmark spike](specs/competitive-benchmarks.md); **[Metal dispatch attribution](specs/metal-dispatch-attribution.md)**; [Metal/MLX reuse study](specs/metal-mlx-reuse-study.md) | `ACTIVE` | `CLAIM-BACKEND-METAL-M3B-1` | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index d47ff89a7..374f1be80 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -274,7 +274,7 @@ evidence. |---|---|---|---|---| | `BACKEND-CUDA-SM121` | GB10/sm121a | `PARTIAL` | gate workload built, traced, token/perf gated; full component-family coverage is open | [backend row](backend-matrix.md#cuda-target-rows) | | `BACKEND-CUDA-OTHER` | vLLM sm70/75/80/86/87/89/90/100/101/103/110/120 targets | `ACTIVE` | 9 CUDA arches build-supported (sm80/86/87/89/90a/100a/103a/110/120a, single-arch portable-kernels-only, `-Werror` clean, SASS emitted); sm70/75/101 not build-supported; no non-121a target is runtime-validated | [backend matrix](backend-matrix.md), [CUDA inventory](specs/cuda-architecture-inventory.md) | -| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 R4-R5 is green: QEMU-built artifact, exact operation/model output, and an AAPCS64 Q8 leaf that beats compiler SDOT 3.66-5.08% on M1/T1 and M128 while improving full-model E2E 0.13%. M1/T4 is −2.43%; Pi memory/concurrency and same-file llama.cpp floor remain open | [backend matrix](backend-matrix.md), [A76 Q8 dot](specs/cpu-a76-q8-dot.md), [binding evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md) | +| `BACKEND-CPU` | production CPU | `PARTIAL` | shared CPU path is correctness-gated and the 20-core Arm/i8mm Qwen3.5-2B single-stream llama.cpp floor is closed; server concurrency is open. Raspberry Pi 5 Cortex-A76 R4-R5 is green: QEMU-built artifact, exact output, and an AAPCS64 Q8 leaf that beats compiler SDOT 3.66-5.08% on M1/T1 and M128. Its separate same-file llama.cpp floor is now measured/NOT MET on speed: vllm.cpp is 0.461x prefill and 0.653x decode/E2E, while using 24.2% less RSS; exact-prompt 64-token output matches. M1/T4 is −2.43%; BF16 GEMM, speed closure and concurrency remain open | [backend matrix](backend-matrix.md), [A76 Q8 dot](specs/cpu-a76-q8-dot.md), [assembly evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md), [llama.cpp evidence](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md) | | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) | | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) | | `BACKEND-VULKAN` | Vulkan | `INVENTORIED` | runtime absent | [backend matrix](backend-matrix.md) | diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index 0b566d661..10ae2e205 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -112,7 +112,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner | |---|---|---|---|---|---|---|---| -| `KERNEL-CPU-A76-Q8-DOT` | Cortex-A76 Q8_0 x Q8_0 SDOT, exact-order C++ and scheduled AAPCS64 variants for the Pi decode/prefill keep-quant GEMM | llama.cpp `ggml/src/ggml-cpu/quants.c:400` portable dot and `arch/arm/quants.c:1076-1160` Arm DotProd @ project pin `237ad9b96`; Qwen semantics remain vLLM `555967922` | [ACLE SDOT + HWCAP/MIDR selector](../src/vt/cpu/cpu_quant_dot_sdot.cpp#L85), [AAPCS64 two-block leaf](../src/vt/cpu/cpu_quant_dot_a76.S#L10), [Q8 dispatch](../src/vt/cpu/cpu_quant_dot.cpp#L675), [same-binary variants](../examples/cpu_kernel_bench/main.cpp#L858) | [direct exact variant/edge tests](../tests/vt/test_ops_quant_dot.cpp#L593); QEMU 20/20, 150258 assertions; physical Pi exact checksums/tokens; assembly vs compiler SDOT +3.66% M1/T1, +5.08% M128/T1, +3.69% M128/T4, but −2.43% M1/T4; [binding evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md#L40) | [A76 Q8 dot spec](specs/cpu-a76-q8-dot.md) | `GATING` | - | +| `KERNEL-CPU-A76-Q8-DOT` | Cortex-A76 Q8_0 x Q8_0 SDOT, exact-order C++ and scheduled AAPCS64 variants for the Pi decode/prefill keep-quant GEMM | llama.cpp `ggml/src/ggml-cpu/quants.c:400` portable dot and `arch/arm/quants.c:1076-1160` Arm DotProd @ project pin `237ad9b96`; Qwen semantics remain vLLM `555967922` | [ACLE SDOT + HWCAP/MIDR selector](../src/vt/cpu/cpu_quant_dot_sdot.cpp#L85), [AAPCS64 two-block leaf](../src/vt/cpu/cpu_quant_dot_a76.S#L10), [Q8 dispatch](../src/vt/cpu/cpu_quant_dot.cpp#L675), [same-binary variants](../examples/cpu_kernel_bench/main.cpp#L858) | [direct exact variant/edge tests](../tests/vt/test_ops_quant_dot.cpp#L593); QEMU 20/20, 150258 assertions; physical Pi exact checksums/tokens; assembly vs compiler SDOT +3.66% M1/T1, +5.08% M128/T1, +3.69% M128/T4, but −2.43% M1/T4. Same-file llama.cpp floor measured/NOT MET: vllm.cpp 0.461x prefill / 0.653x decode+E2E, 0.758x peak RSS (better), exact-prompt output equal; [assembly evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md#L40), [competitor evidence](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md) | [A76 Q8 dot spec](specs/cpu-a76-q8-dot.md) | `GATING` | - | | `KERNEL-ACCEL-PROVIDER-SELECT` | **WHICH implementation of an op runs, when more than one exists on a device** — the selection layer above every kernel family in this matrix. Distinct from `KERNEL-CUDA-DISPATCH-AOT`, which is about which ARCH a CUDA kernel is compiled/selected for; this is about which PROVIDER (ours, a vendor library, MLX, llama.cpp) serves the op at all | no single upstream file: this is the shape vLLM's runtime chain uses everywhere — flashinfer tactic registries, cuBLASLt/CUTLASS per-call heuristics, and torch's backend selection — rather than compile-time pinning | `vt::OpProvider` [op_provider.h](../include/vt/op_provider.h) + [op_provider.cpp](../src/vt/op_provider.cpp); the flat `[OpId][DeviceType]` `void*` table it replaces is gone from [ops.cpp](../src/vt/ops.cpp) with the ~70 op wrappers untouched. Providers registered today: `vt-native` (every backend kernel in the tree, priority 0, unconditional — behaviour preserved exactly), `mlx` (priority 100, Metal `kMatmul`/`kMatmulBT`, build-gated `VLLM_CPP_MLX`, [metal_mlx_provider.mm](../src/vt/metal/metal_mlx_provider.mm)), and — NEW 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`, work row `S5`) — **`vt-cpu-ref` (priority −1000, the portable reference tier)**: the CPU kernel installed LAZILY as a negative-priority fallback on a UNIFIED-MEMORY device's first `GetOp` miss, mirroring `custom_op.py:138 forward_native`, so a partial backend runs an op it lacks natively instead of throwing. Native always wins (priority); gated on `Backend::UnifiedMemory()` (a discrete GPU never gets it — a CPU kernel on true device memory is corruption); observable via `GetReferenceTierHits()` + a one-time loud stderr line | [test_op_provider.cpp](../tests/vt/test_op_provider.cpp) 11 cases / 47 assertions — deterministic selection under REVERSED registration order, name tie-break, duplicate rejection, capability predicate, caps re-resolution, decline-and-fall-back, stats, runtime disable; [test_metal_backend.cpp](../tests/vt/test_metal_backend.cpp) 9 cases / 108 assertions on the M4 with MLX ON, including MLX-vs-MSL-vs-CPU NMSE per op at real shapes and an end-to-end DECLINE; **[test_reference_tier.cpp](../tests/vt/test_reference_tier.cpp) (S5): discrete-device refusal + unified-device zero-native-kernel fallback correctness + native-wins + observability, hardware-free via a fake backend on `kXPU`.** Linux CPU 156/156; dgx regression set ALL UNCHANGED — anchor `tests/vt/test_op_provider.cpp:64` | [Metal/MLX reuse study §6](specs/metal-mlx-reuse-study.md); [accelerator-seam-audit §10](specs/accelerator-seam-audit.md); [drop-in kernel ABI](specs/dropin-kernel-abi.md) (the complementary ARGUMENT half) | `ACTIVE` — mechanism landed and gated with THREE provider kinds (`vt-native`, `mlx`, `vt-cpu-ref`); the CUDA/CPU/Vulkan vendor provider rows it was designed for are not yet populated (so the row is deliberately left open) | `CLAIM-BACKEND-ACCEL-PROVIDER-1` | | `KERNEL-CUDA-DISPATCH-AOT` | CUDA runtime, streams, graphs, per-arch dispatch, and generated/AOT artifact selection | platform selection `vllm/platforms/cuda.py:205-493`; gencode handling `CMakeLists.txt:201-220`; JIT packages `cmake/external_projects/triton_kernels.cmake:1-28` | [cuda_backend.cu:20](../src/vt/cuda/cuda_backend.cu#L20), [CMakeLists.txt:37](../CMakeLists.txt#L37), [TritonAOT.cmake:57](../cmake/TritonAOT.cmake#L57) | [CUDA backend tests](../tests/vt/test_cuda_backend.cpp#L31); the Triton GDN AOT tree is now vendored per-arch — `sm_121a` (RUNTIME-VERIFIED) plus `sm_80/86/89/90a/100a` (DERIVED+BUILD-VERIFIED, `CLAIM-TRITON-AOT-PER-ARCH`, 2026-07-28), each a real per-target cubin (`cuobjdump sm=80/86/89/90/100`); the CUDA fast-path arch fan-out (non-GDN) remains as evidenced in `backend-matrix.md` | [inventory](specs/kernel-family-inventory.md); [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `ANCHOR-BACKFILL` | - | | `KERNEL-FUSION-FRAMEWORK` | **Portable automatic op-fusion framework (roadmap_v1 ORDER-1 extensibility cornerstone)** — declare-once `constexpr FusedRecipe` catalog (backend-agnostic, above `vt::`) transcribing vLLM's finite fusion-pass set; realize-per-backend through the `vt::` op table (Tier-0 composite = CPU oracle inherited free; Tier-1 interpreter = one kernel/backend lights up every recipe). A new vLLM fusion PR = ONE declaration; a new GPU = one catalog realization; a new model = additive pattern bindings. NOT a perf lever (35B ceiling ~3.5%/step compute-bound); primary value = extensibility + mechanical upstream-sync + CPU/CUDA oracle-drift elimination | pass manager `vllm/compilation/passes/pass_manager.py:86,138-200`; the finite fusion-pass set `passes/fusion/{rms_quant_fusion.py:618,act_quant_fusion.py:283,qk_norm_rope_fusion.py:188,attn_quant_fusion.py:362}`; declare-once analog `vllm/model_executor/custom_op.py:103` (`forward_native`/`forward_cuda`/`enabled()` :138-207); combo_kernels OUT-OF-SCOPE `vllm/config/compilation.py:967` | Phase-0 skeleton LANDED: [fused_recipe.h:72](../include/vt/fused_recipe.h#L72), [recipes.h:35](../include/vt/recipes.h#L35), [kFusedChain dispatch](../src/vt/ops.cpp#L639), Tier-0/1 CUDA [cuda_ops.cu:1062](../src/vt/cuda/cuda_ops.cu#L1062) + CPU [cpu_ops.cpp:1427](../src/vt/cpu/cpu_ops.cpp#L1427); catalog-to-be = the hand-fused ops (`kRmsNormQuantFp8`/`kSiluMulFp4Quant`/`kAttnQkNormRopeGate`/`kSigmoidGateFp4Quant`/`kRmsNormGatedQuantFp8`). **2026-07-19 (`CLAIM-FUSION-FRAMEWORK-W0`): W0 ADOPTED at ONE production site** — `RunLayerPaged` post_attention_layernorm (35B MoE paged, `test_qwen36_paged_engine` hot path) now routes the plain add+residual+gemma-RMSNorm through [`vt::FusedChain(kFusedAddRmsNorm)`](../src/vllm/model_executor/models/qwen3_5.cpp#L4784) behind [`FusedChainAdoptEnabled()` (`VT_FUSED_CHAIN_ADOPT`, default ON / `=0` restores the exact prior `vt::RmsNorm(residual)` hand-call)](../src/vllm/model_executor/models/qwen3_5.cpp#L1394). BEHAVIOR-PRESERVING by construction (Tier-0 composite dispatches to the same `vt::RmsNorm(residual)` primitive; perf-neutral, NOT a speedup) — proves the declare-once/realize-per-backend seam end-to-end in production. The other 3 identical plain sites (`RunLayer`/`RunDenseLayer`/`RunDenseLayerPaged` post-attn + the input-layernorm plain path) are left on the hand-call; W1's generalized POD (multi-input operands + quant/silu/rope opcodes) is required only for the quant-fused sites (`kSiluMulFp4Quant`/`kRmsNormQuantFp8`/…), NOT for these plain add+rmsnorm sites — the current 3-opcode/4-role POD expresses them byte-identically. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W1`): W1 GENERALIZED the POD (infrastructure only, perf-neutral, NO call site changed).** `FOp` grew from `{kAdd,kMul,kRmsNorm}` to add `kSilu`/`kSigmoid` (Tier-1 elementwise) + fused-primitive opcodes mapping 1:1 to standalone ops so the composite is byte-exact by construction — `kSiluMul`→`vt::MoeSiluMul`, `kSigmoidGate`→`vt::SigmoidGateBf16`, `kRmsNormGated`→`vt::RmsNormGated`, `kRope`→`vt::RopeFromCache`, `kQuantFp8`→`vt::QuantFp8Static`, `kQuantFp4`→`vt::ScaledFp4Quant`, `kAttnQkNormRopeGate`→`vt::AttnQkNormRopeGate`. The fixed `{kIn,kResidual,kWeight,kOut}` 4-role model became an INDEXED `FOperandSlot operands[8]` table (`FStep` names input indices + `out`/`out2`); tensors bind positionally via `FusedBinding`, scalars via `FusedParams`; intermediates are caller-bound scratch (byte-exact to the unfused sequence). Tier-0 composite is now ONE device-agnostic walker in `ops.cpp` (per-opcode dispatch to the standalone op, self-dispatching per device — eliminates CPU/CUDA oracle drift); the per-backend `kFusedChain` op is now ONLY the Tier-1 interpreter over the canonical `[x,weight,residual,out]` shape. All five W2 target chains DECLARED as `constexpr` recipes ([recipes.h](../include/vt/recipes.h), citing each vLLM pass). TIER (honest): composite realizes all five byte-exact; Tier-1 covers only the elementwise/rmsnorm subset (W0 `kFusedAddRmsNorm`) — the four quant/activation chains + attn preamble are COMPOSITE-ONLY. Backend-negotiation (§3b/§6): fp8 terminal `vt::QuantFp8Static` CUDA-only (fp8 chains' composite CUDA-only; CPU covers up to the bf16 norm); `kAttnQkNormRopeGate` a composite-only MACRO (per-head 3-D operands → dispatches the whole preamble to the single `vt::AttnQkNormRopeGate` op). ALL FIVE byte-exact expressible, none forced. Anchors: generalized POD [fused_recipe.h](../include/vt/fused_recipe.h), binding/params/general entry [ops.h](../include/vt/ops.h), device-agnostic composite [`FusedChainComposite`](../src/vt/ops.cpp), Tier-1 interpreters [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)+[cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W2`): W2 MIGRATED the bespoke hand-fusions to `vt::FusedChain(recipe)` — the framework now OWNS the fusion dispatch, perf-neutral by construction.** New realization-binding infra (the "fast realization" tier the spike §3c reserved): `FusedRecipe.fast_op` ([fused_recipe.h](../include/vt/fused_recipe.h)) carries the raw `OpId` of the EXISTING single-launch bespoke fused kernel that realizes the whole recipe (backend-AGNOSTIC — names WHICH abstract op, realized per-backend via the op table); a non-throwing `OpRegistered` probe + a per-recipe `DispatchFusedFast` adapter switch + convenience overloads in [ops.cpp](../src/vt/ops.cpp)/[ops.h](../include/vt/ops.h). `FusedChain(recipe,…)` dispatches recipe→fast_op WHEN the backend registers that OpId (the SAME kernel the model called directly pre-migration → byte-identical AND perf-neutral; no extra kernel, no per-forward getenv/alloc), else falls through to the byte-exact Tier-0 composite (graceful degradation). The composite is the exposed oracle ([`FusedChainComposite`](../src/vt/ops.cpp)) the fast realization is validated against. Bound recipes: `kRmsNormQuantFp8`→`OpId::kRmsNormQuantFp8`, `kRmsNormGatedQuantFp8`→`OpId::kRmsNormGatedQuantFp8`, `kSiluMulFp4Quant`→`OpId::kSiluMulFp4Quant`, `kSigmoidGateFp4Quant`→`OpId::kSigmoidGateFp4Quant` ([recipes.h](../include/vt/recipes.h)); `kAttnQkNormRopeGate` has NO fast_op (its composite MACRO already dispatches to the single `vt::AttnQkNormRopeGate` launch — perf-neutral trivially). SIX call sites migrated in [qwen3_5.cpp](../src/vllm/model_executor/models/qwen3_5.cpp) behind `VT_FUSED_CHAIN_ADOPT` (`=0` restores the exact prior hand-calls, same binary): `kSiluMulFp4Quant` (MoE down-proj), `kSigmoidGateFp4Quant` (full-attn o-proj), `kRmsNormGatedQuantFp8` ×2 (GDN out-proj), `kRmsNormQuantFp8` (input-layernorm), `kAttnQkNormRopeGate` ×2 (attn preamble). qwen3_5.cpp net +61 lines (−21/+82): each migrated fused-op call became one `FusedChain` call (call-for-call neutral), but the mandated same-binary rollback `else` branch + explanatory comment at each of the 6 sites grows the raw count — the true shrinkage lands when the rollback branches and bespoke `OpId`s retire; the dispatch surface is already framework-OWNED. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W3`): W3 MECHANICAL-UPSTREAM-SYNC PROOF** — ported a NEW, previously-unported vLLM fusion-pass variant, `SiluMulFp8StaticQuantPattern` (`vllm/compilation/passes/fusion/act_quant_fusion.py:81` → `_C.silu_and_mul_quant`; registered unconditionally at `act_quant_fusion.py:296`), as ONE `constexpr FusedRecipe kSiluMulQuantFp8` ([recipes.h](../include/vt/recipes.h)) + its byte-exact test. The static-per-tensor-FP8 sibling of `kSiluMulFp4Quant`; its Tier-0 composite is realized ENTIRELY from EXISTING standalone `vt::` ops (`kSiluMul`→`vt::MoeSiluMul`, then `kQuantFp8`→`vt::QuantFp8Static`) — NO new primitive, NO kernel, NO composite-walker case, NO dispatch edit, NO model-site edit. `fast_op = kNoFastOp` (no bespoke silu·mul→static-fp8 fused OpId exists in-tree; a fast kernel is a separate later perf step). **Additivity headline: adding a whole new fusion pattern touched exactly 2 shared files** — `include/vt/recipes.h` (+1 declaration) + `tests/vt/test_ops_fused_chain.cpp` (+1 byte-exact test); `git diff --stat main` = those 2 files only. This IS the framework's primary-value proof (spike §4/§10): a new upstream fusion PR = one declaration. **2026-07-20 (`CLAIM-FUSION-FRAMEWORK-W4`): W4 — the EXECUTABLE BACKEND-ADDITIVITY PROOF, closing the W-series.** Makes the claim executable: "a new backend registers `kFusedChain` (Tier-0 composite) ONCE and inherits the ENTIRE catalog correct, ZERO per-recipe work." Proof approach (spec §10 W4, PREFERRED option b): treat the EXISTING CPU backend AS the 'second backend' relative to CUDA — NO mock `DeviceType` (which would edit the core enum + every switch, ironically non-additive). New test [`test_fused_chain_additivity.cpp`](../tests/vt/test_fused_chain_additivity.cpp) enumerates the WHOLE catalog (`kCatalog`, all 7 recipes) and in ONE generic loop asserts each runs correct on the CPU backend via the Tier-0 composite ([`vt::FusedChainComposite`](../src/vt/ops.cpp)), BYTE-EXACT vs the standalone-op-sequence golden, over the CPU-expressible scope: 4 CPU-FULL end-to-end (`kFusedAddRmsNorm`/`kSiluMulFp4Quant`/`kSigmoidGateFp4Quant`/`kAttnQkNormRopeGate`), 3 CPU-PREFIX (`kRmsNormQuantFp8`/`kRmsNormGatedQuantFp8`/`kSiluMulQuantFp8` — static-fp8 terminal `vt::QuantFp8Static` CUDA-only per §3b/§6; the prefix is byte-exact and the FULL composite is asserted to THROW on CPU, documenting the backend-negotiated tail, not silently skipping). ADDITIVITY EVIDENCE (catalog GROWS ⇒ backend does NOT): catalog [recipes.h](../include/vt/recipes.h) 1→6→7 recipes (W0→W1→W3); the composite walker [`FusedChainCompositeImpl`](../src/vt/ops.cpp) is ONE function with a per-OPCODE switch (12 `FOp::` cases, NOT per-recipe); the CPU/CUDA `kFusedChain` registration is ONE line each; `src/vt/cpu/cpu_ops.cpp` does NOT `#include recipes.h` (the backend never sees the catalog); W3's whole new recipe `kSiluMulQuantFp8` appears in ZERO backend TUs + ZERO in the `ops.cpp` dispatch — it runs purely through the shared walker (that is WHY W3 touched 2 files and no `src/vt/`). **KERNEL-FUSION-FRAMEWORK ORDER-1 milestone DONE** (W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity). HONEST DEFERRED / future (named, non-blocking the ORDER-1 milestone): (a) the Tier-1 PERF interpreter covers only the elementwise/rmsnorm subset — the quant/activation chains realize composite-only (correct/byte-exact, but the single-pass fused perf kernel for the quant chains is future); (b) a REAL Metal/Vulkan realization of the catalog needs the M4 dev-box (HW-blocked); (c) per-recipe fast single-launch kernels for composite-only recipes are separate later perf steps. Perf is NOT the framework's value (§11: 35B ceiling ~3.5%/step, compute-bound). **2026-07-25 (`CLAIM-FUSION-CONSISTENCY-AUDIT`): CONSISTENCY AUDIT + ENFORCEMENT** — read-only static audit of catalog USAGE across all 12 model forwards ([fusion-consistency-audit.md](specs/fusion-consistency-audit.md)). Verdict MOSTLY consistent: qwen3_5 family fully migrated; qwen3/qwen3_moe/deepseek_v2 adopted `kFusedAddRmsNorm{,Std}`; **DRIFT — gemma/gemma2/gemma3/glm4/phi3 hand-call the residual `vt::RmsNorm(...,&res)` add+RMSNorm chain and never route through the catalog** though the recipe already exists (one-line adoption each = follow-on `FUSION-DENSE-MIGRATE`). olmo2/granite/opt deliberately-not-fused (post-norm/LayerNorm). GDN glue + MoE-combine remain bespoke (spike §10 W2 remainder). NEW enforcement CI check [`scripts/check-fusion-consistency.py`](../scripts/check-fusion-consistency.py) (+ mutation test, wired into the `agent-record` CI job) flags any model that hand-fuses add+RMSNorm without the catalog unless allowlisted — the 5 drift models on [`fusion-consistency-allowlist.txt`](../scripts/fusion-consistency-allowlist.txt) keep the gate GREEN while blocking new silent bypasses; removing a stem is the gate closing. | [test_ops_fused_chain.cpp](../tests/vt/test_ops_fused_chain.cpp) (byte-exact composite==interp==golden; W1 extends it: each new recipe's Tier-0 composite == its standalone-op-sequence golden byte-exact, fp4+attn CPU+CUDA, fp8 CUDA-only. **W2 adds the fast==composite==golden triple arm per recipe — CPU 228 assertions, CUDA-on-dgx 420, memcheck 0**). **W3 adds `RunSiluMulQuantFp8Cuda` (kSiluMulQuantFp8: FusedChain==composite==MoeSiluMul+QuantFp8Static golden, byte-exact, CUDA — the newly-ported pass's test), and the no-token-regression gate `test_qwen27_paged_engine` 235/235 + `test_qwen36_paged_engine` 315/315 unchanged (the new recipe is DECLARED, not wired into any model — the engine is untouched)**; **W4 adds `test_fused_chain_additivity.cpp`** (the whole-catalog additivity loop: every recipe byte-exact on the CPU 'second backend' via the Tier-0 composite, CPU-full end-to-end / fp8-prefix + negotiated-tail-throw; count guard `==7`) — **1 case / 17 assertions CPU green, sibling `test_ops_fused_chain` 228/228 unchanged, engine byte-identical so 27B 235/235 + 35B 315/315 structurally unchanged**; to port: `tests/compile/passes/test_fusion.py:249-287` oracle discipline (byte-exact, stricter than upstream `assert_close`) — anchor `tests/vt/test_ops_fused_chain.cpp:124` | [portable-fusion-framework.md](specs/portable-fusion-framework.md) | `ANCHOR-BACKFILL` (W-series ORDER-1 PROOF MILESTONE DONE: W0 adopt + W1 POD + W2 migrate + W3 mechanical-sync + W4 backend-additivity, all merged+gated; row stays SPIKE-open because the framework retains named future work — the Tier-1 quant-chain perf interpreter (composite-only today) + a real Metal/Vulkan catalog realization (M4 HW-blocked) + per-recipe fast kernels, §10 Wn) | `CLAIM-FUSION-FRAMEWORK-W4` | diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index f27069879..e25174db6 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -916,3 +916,4 @@ Columns: | 2026-08-06 (`row/H3-FP4-SPEED`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; lifecycle unchanged) | **MiniMax-H3 W-FP4a — fp4-RESIDENT NVFP4 routing for the device DiT forward (NO new quant code).** Until now both NVFP4 loaders dequantized the packed FP4 projections to bf16 and the device forward ran `vt::MatmulBT`, so the sm_121a FP4 tensor-core route never ran for H3. Adds `Nvfp4Weight` carriers to `MiniMaxH3DitBlockWeights`/`MiniMaxH3DitWeights`, a fp4-resident streamer `StreamMiniMaxH3Nvfp4ToDeviceFp4` (keeps the compressed-tensors triple host-resident; the shared dispatcher uploads+repacks lazily then frees the fp4 originals, so peak device memory is ~1/4 of the bf16 arm), and a `LinearDev` dispatch that routes a non-Empty fp4 projection through `dense_nvfp4::MatmulNvfp4W4A16D`. | The routing is vLLM's OWN forced-Marlin-for-a16 selection: the checkpoint is weight-only NVFP4 (no `input_activations`, `IsTrueW4A4()==false`), so `kernels/linear/__init__.py:879-881` forces the Marlin W4A16 kernel, mirrored by `include/vllm/model_executor/models/dense_nvfp4_gemm.h:12-22,505-549` (`MatmulNvfp4W4A16D` -> single-expert `vt::MoeGroupedGemmNvfp4Marlin`, the SAME kernel Laguna routed-experts + dense Qwen3-32B NVFP4 use). Not cutlass-fp4/W4A4 (needs fp4 activations, private to `qwen3_5.cpp`). fc1 is already merged `[gate;up]` -> one W4A16 GEMM + `vt::SiluAndMul`. | **CPU-GATED (wiring), verified**: `test_minimax_h3` 62/62 cases / 30039 assertions, 0 failed. The synthetic-NVFP4 case streams the fp4 twin, asserts the loader kept the projections PACKED (fp4 slot set / bf16 slot Empty, and the inverse for the bf16 loader), runs fp4 + bf16 device forwards on the SAME file, asserts the W4A16 dispatcher executed ALL 11 quantized GEMMs (the `Nvfp4W4A16Stats` this-path-ran counter), and bounds fp4-vs-bf16 <= 2e-3. On CPU the dispatcher has no Marlin op so it falls to the bf16 arm's own dequant+matmul (hence a WIRING gate here); the Marlin kernel numerics are CUDA-gated independently by `test_ops_nvfp4_matmul` / `test_linear_method` (2e-3 f32-out / 8e-3 bf16-out vs a bf16 reference). `benchmark_binding=false`. PENDING: GB10 CUDA build + the fp4-vs-bf16 numeric delta and steady per-step timing (disk/build window); real-checkpoint t2va e2e DISK-BLOCKED (~41 GB working set). Comparability: vLLM-Omni serves NO quantized H3 (BF16-only in practice; source-audited `a4ea67a2`, spec §8.3) -> HW/loader-forced-indirect. | | 2026-08-06 (`row/BACKEND-CPU`; `BACKEND-CPU` R1; PR #65; lifecycle remains `PARTIAL`) | Adds `vllm-cpu-kernel-bench`, a developer-only vt-op benchmark substrate: deterministic quant-GEMM fixtures, calibrated batched timing, cache-pressure profiles, affinity, JSON, checksums, system metadata, and grouped generic/Cortex-A76 `perf_event_open` counters with explicit multiplex/unsupported status. Production dispatch and numerics are unchanged. | No vLLM behavior counterpart; vLLM remains the x86 semantic oracle and llama.cpp `237ad9b96` remains the Pi performance floor. The quant fixture invokes the existing `vt::MatmulBTQuant` contract unchanged. | **CPU-GATED, `benchmark_binding=false`.** GCC 15.2 `-Wall -Wextra -Werror` build; clang-format clean; `test_cpu_kernel_bench_cli` deterministic JSON schema/checksum + invalid-input + structured-counter cases; direct 1/4-thread x86 runs and real generic PMU counts. X86 timings are non-binding tool validation. Pi PMU execution, model correctness, throughput and memory all remain `PENDING`. | | 2026-08-06 (`KERNEL-CPU-A76-Q8-DOT` R4-R5; `CLAIM-KERNEL-CPU-A76-Q8-DOT`; physical RPi5 Cortex-A76; closing commit: this checkpoint) | Adds an exact-order ACLE SDOT control and an original AAPCS64 two-block Q8_0×Q8_0 leaf behind Linux DotProd/MIDR dispatch. `auto` selects assembly only on Cortex-A76+DotProd; x86, non-DotProd and other Arm CPUs retain portable dispatch. Explicit `portable`/`sdot`/`a76-asm` same-binary controls remain. ARM64 builds/tests locally through buildx/QEMU; Pi is execution-only. | The integer structure is informed by llama.cpp `237ad9b96` `ggml/src/ggml-cpu/arch/arm/quants.c:1076-1160`, while this port deliberately retains the local portable function's per-block f32 reduction order. vLLM `555967922` supplies Qwen3.5 semantics, not a corresponding CPU microkernel. Local anchors: `src/vt/cpu/cpu_quant_dot_{sdot.cpp,a76.S}`, Q8 dispatch in `cpu_quant_dot.cpp`, direct tests in `tests/vt/test_ops_quant_dot.cpp`, and [immutable evidence](../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). | **PASS for the compiler-gap/component gate; row `GATING`, `benchmark_binding=true`.** Final binaries `9eb57cf...`/`a94dad30...`; QEMU focused suite 20/20, 150258 assertions; physical-Pi checksums exact. Assembly vs compiler SDOT wall/cycles/instructions: M1/T1 +3.66%/+3.17%/+10.10%, M128/T1 +5.08%/+4.61%/+10.24%, M128/T4 +3.69%/+3.69%/+9.74%. M1/T4 is an explicit −2.43% wall/−4.32% cycles residual despite 8.77% fewer instructions. All 64 Qwen tokens equal the x86 golden in all nine runs; median assembly vs SDOT TTFT −1.55%, TPOT −0.05% neutral, E2E −0.13%. Disassembly proves GCC's framed dependent one-block loop versus the stack-free independent two-block schedule. Same-file Pi llama.cpp, peak memory and concurrency remain `PENDING`; no competitor-floor binding is claimed. | +| 2026-08-06 (`KERNEL-CPU-A76-Q8-DOT` R6 competitor checkpoint; physical RPi5 Cortex-A76; lifecycle remains `GATING`; closing commit: this checkpoint) | Measures the separate four-core A76 same-file llama.cpp floor after the assembly leaf became default. No production code changes. The vllm.cpp nominal p16 request measures 17 input tokens, so the binding competitor uses pp17/tg64/pp17+tg64. A same-text CLI arm verifies 64-token greedy output equality. | Official llama.cpp tag b9892 `ee445f93d` reconstructed under QEMU because historical recorded fork object `237ad9b96` is unavailable; exact recorded anchors match (`quants.c:400`, `arch/arm/quants.c:1076`, `repack.cpp:2725`, `qwen35.cpp`). Local evidence: [Pi competitor record](../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md). | **CORRECTNESS PASS, PERFORMANCE NOT MET, `benchmark_binding=true`.** Three clean unthrottled vllm.cpp reps: prefill 12.81 tok/s, decode 2.55 tok/s, output-equivalent E2E 2.46 tok/s, E2E 26,018.39 ms. llama.cpp three-sample p17/tg64/combined: 27.77 / 3.91 / 3.77 tok/s, E2E 16,998.49 ms. vllm.cpp ratios 0.461x prefill / 0.653x decode+E2E; peak RSS wins 2.841 vs 3.747 GiB (24.2% less). Same-text normalized output SHA `a5a630d7...` equal; all vllm performance tokens retain golden SHA `0ec98e...`. Intrusive 50 ms forked sampler run VOID; accepted timing has no sampler, RSS sampled separately at 1 Hz. Next lever: fresh both-engine profile, then BF16 GEMM; M1/T4 and concurrency remain. | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 9ffb1520d..6469fea93 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -68,7 +68,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 7 | `ROAD-V1-C7` | Sampling/API controls and logprobs payloads | [engine matrix](engine-matrix.md), [coverage view §6](feature-matrix.md#6-sampling--generation-controls), [C7 spec](specs/sampling-controls-c7.md) | **W1-W4 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7`, NOT pushed): the full sampling-control surface is WIRED end-to-end + gated exactly on the CPU reference backend.** `SAMPLE-CORE` + `SAMPLE-LOGIT-FILTERS` -> `ACTIVE`: SamplingParams undefers + validates logit_bias/allowed_token_ids/bad_words + all_stop_token_ids; OpenAI protocol parses the three fields + clamps logit_bias to [-100,100]; InputBatch tracks + `build_sampling_metadata` populates min_p/min_tokens/logit_bias/allowed_token_ids_mask/bad_words/max_num_logprobs with condense/swap maintenance; InputProcessor tokenizes bad_words + seeds the eos stop-set. RED-first (disabling a wiring line fails the gate); inertness proven (default/greedy build byte-identical SamplingMetadata, SACRED greedy unaffected); CPU `-Werror` 0-warn. The transforms are device-neutral pure functions gated exactly (not near-tie). **W5 LANDED + CPU-GATED (2026-07-27, `CLAIM-ROADMAP-C7-LOGPROBS`, NOT pushed): the logprobs PAYLOAD is wired end-to-end + gated on the CPU reference engine.** `SAMPLE-LOGPROBS` -> `DONE`: ported `vllm/logprobs.py` + `vllm/v1/engine/logprobs.py` (LogprobsProcessor); threaded ModelRunnerOutput.logprobs -> scheduler slice_request -> EngineCoreOutput.new_logprobs -> OutputProcessor -> CompletionOutput.logprobs -> OpenAI `CompletionLogProbs`/`ChatCompletionLogProbs` serialization (grounded 1:1 in vLLM 0.26 `555967922`). Gate: serialization vs a hand-computed vLLM oracle (completion N+1 vs chat N cutoff, bytes, text_offset, -9999 floor) RED-first, + e2e through the CPU Qwen3.6 engine (`logprobs=K` greedy sampled==argmax==rank1, chat `top_logprobs`, null-when-off inertness); CPU `-Werror` 0-warn. **C7-core (SAMPLE-CORE + SAMPLE-LOGIT-FILTERS + SAMPLE-LOGPROBS) is complete.** | `DONE` | **REMAINING (separate rows, NOT required for C7-core):** `SAMPLE-PROMPT-LOGPROBS` payload/serialization is done but its runner prompt-position-logits SOURCE is a runner/prefill addition (row `PARTIAL`); plus INVENTORIED `n>1` execution, `SAMPLE-PHILOX` exact RNG, `SAMPLE-LOGPROB-TOKEN-IDS`/logprobs_mode, beam search. **Custom logits processors LANDED 2026-07-28 (`SAMPLE-CUSTOM-PROCESSORS` -> `ACTIVE`, `CLAIM-C7-CUSTOM-LOGITS`): per-request C-ABI callback (`vllm_logits_processor`, ABI v8) at the sampler's non-argmax-invariant stage; default byte-identical; also closes SGLang `SGLANG-SAMPLING-CUSTOM`.** **Parallel sampling `n>1` LANDED 2026-07-28 (`SAMPLE-N` -> `ACTIVE`, `CLAIM-C7-N-SAMPLING`): the OpenAI `n` parameter now EXECUTES — an `n>1` request fans out into n prompt-sharing child sequences (`ParentRequest`/`FanOutParallelSampling`, mirroring vLLM `parallel_sampling.py`) aggregated into one `RequestOutput` / n indexed `choices`; `n==1` byte-identical; CPU-gated RED-first (1→n) with a `top_k=1` determinism gate. Residuals: `best_of`, async-streaming per-child collation, C-ABI `n`.** **Beam search LANDED 2026-07-28 (`SAMPLE-BEAM` -> `ACTIVE`, `CLAIM-C7-BEAM`): beam search as an OUTER engine loop (mirroring vLLM `entrypoints/generate/beam_search/{utils,offline}.py`) — per step one decode per beam (`logprobs=2*beam_width`), expand + score by `get_beam_search_score` (cum_logprob / `seq_len**length_penalty`), keep top-`beam_width`, retire EOS, return `beam_width` outputs (reuses the `SAMPLE-N` multi-output seam). Deterministic ⇒ token-EXACT: model-free gate vs a hand-computed beam tree (RED-first 1→2 beams) + e2e over the CPU engine (`bw=1`==greedy). Residuals: OpenAI `use_beam_search`/`best_of` wiring, C-ABI beam params, grammar-constrained beams.** **`best_of` + `use_beam_search` OpenAI-ENDPOINT surface LANDED 2026-07-28 (`SAMPLE-BEST-OF` -> `ACTIVE` + `SAMPLE-BEAM` endpoint, `CLAIM-C7-BESTOF-BEAM-API`): both request types gain `best_of`/`use_beam_search`/`length_penalty` + `to_beam_search_params`; `best_of>n` fans out via `SAMPLE-N` and returns the top-n by cumulative logprob (`SelectBestOf`); `use_beam_search` routes through the merged `BeamSearch` driver over the SYNC `LLMEngine` seam and returns `beam_width` beams as choices (endpoint output IDENTICAL to the direct driver call, gated completion + chat). Default (no best_of/beam) path byte-identical. HONEST FINDING: vLLM 0.26 exposes `use_beam_search` on the OpenAI server but has DROPPED `best_of` (only a vestigial `BatchChatCompletionRequest` field) — best_of follows classic OpenAI/V0 semantics gated on our own fan-out. Residuals: async-HTTP beam (driver is `LLMEngine&`-based, production server is AsyncLLM), streaming beam (rejected like upstream), C-ABI best_of/beam.** **Async/production beam LANDED 2026-07-28 (`SAMPLE-BEAM` async coverage, `CLAIM-C7-BEAM-ASYNC`): `BeamSearchAsync(AsyncLLM&, …)` mirrors vLLM `entrypoints/generate/beam_search/online.py:28-220`, drives the production AsyncLLM (the engine the HTTP server holds) per-beam single-token `generate` and calls the SAME merged `BeamSearchStep`/scoring (shared template driver; only the engine differs). `use_beam_search` on the server now RUNS instead of raising "requires the synchronous engine"; gated token-IDENTICAL to the sync `BeamSearch` (bw 1/2/3) + endpoint choices IDENTICAL over the AsyncLLM backend (completion + chat). Per-beam decodes driven SEQUENTIALLY (byte-identical to sync); concurrent per-beam stepping (online.py `asyncio.gather`) is a named residual. Remaining beam residuals: streaming beam, C-ABI beam, grammar-constrained beams.** | | 8 | `ROAD-V1-C8` | Tokenize/detokenize, unified streaming parsing and full metrics | [engine matrix](engine-matrix.md), [coverage views §§7,9](feature-matrix.md#7-structured-outputs--tool-calling), [metrics spec](specs/prometheus-metrics.md), [endpoints spec](specs/utility-endpoints.md) | **`SERVE-METRICS` + `SERVE-UTILITY-ENDPOINTS` LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8`, NOT pushed):** the oldest T0 debt — Prometheus `/metrics` (self-contained registry + text-0.0.4 exposition + always-on vLLM metric catalog, names/labels/buckets 1:1) — is CLOSED and gated by the vLLM scrape spec `EXPECTED_METRICS_V1` (RED-first substring gate, `test_prometheus_metrics` 4/4/81); plus `/tokenize`,`/detokenize`,`/ping`,`/server_info`,`/reset_prefix_cache` (additive/opt-in, schema-matched, `test_openai_api_server` 26/26/277). Inertness: opt-in, existing serving byte-identical. **`TOOLS-STREAMING-PARSER` core LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-PARSER`, NOT pushed):** the vLLM 0.26 unified streaming parser ENGINE (declarative `StreamingParserEngine`: token-ID scanner + prefix-buffering lexer + transition state machine + JSON-arg hold-back + drop-info) with qwen3/seed_oss/kimi_k2 configs + the unified registry, EXACT-gated event-for-event vs vLLM 0.26 (`test_streaming_parser_engine` 586/586, RED-first, goldens byte-reproduced from the pin). **Parser ASSEMBLY layer LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-ASSEMBLY`, NOT pushed):** the vLLM 0.26 `ParserEngine` (SemanticEvent -> streaming `DeltaMessage` + one-shot `ExtractedToolCallInformation`) with qwen3/seed_oss/kimi_k2 assembled parsers + `parser_manager` name->parser dispatch, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 9 scenarios, 1652/1652, RED-first 32 asserts, goldens byte-reproduced). **Serving-SSE dispatch swap LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-SERVING`, NOT pushed):** the OpenAI chat streaming path routes engine-backed `--tool-call-parser` names through `parser_manager get_parser_engine` and drives `parse_delta`/`parse`, EXACT chunk-for-chunk vs vLLM 0.26 `chat_completion_stream_generator` (`test_openai_serving_chat_stream` 9 scenarios, 210/210, RED-first 6 CHECKs, goldens byte-reproduced); OFF by default, legacy seam byte-identical (`test_openai_serving` 421/421 unchanged). **5 more engine CONFIG FAMILIES LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS`, NOT pushed):** minimax_m2/glm47_moe/deepseek_v4/deepseek_v32/nemotron_v3 ported as additive `ParserEngineConfig` builders + regex arg-converters + `Glm47MoeParser` name-`.strip()`, field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 19 scenarios, 3510/3510, RED-first 2 asserts; engine-core 586/586 + serving-SSE 210/210 unchanged). **LAST 2 CONFIG FAMILIES gemma4 + inkling LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS-2`, NOT pushed) — vLLM tool-parser family parity CLOSED:** added 4 default-inert assembly-core virtual seams (`preprocess_feed`, virtual `events_to_delta`/`single_pass_parse`/`reset`/`extract_reasoning`, `args_wrapper_keys`) + `gemma4_config`/`inkling_config` (custom key:value + JSON-span arg carvers) + `Gemma4Parser` (channel-injection + `thought\n`-strip) / `InklingParser` (args-key unwrap + trailing-text flush), field-for-field vs vLLM 0.26 (`test_parser_engine_assembly` 26 scenarios, 4526/4526, adds a non-streaming parse() gate, RED-first for all 4 new seams; engine-core 586/586 + serving-SSE 210/210 byte-identical, seams inert). **LIVE PER-STEP METRIC WIRING LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-METRICS-WIRE`, NOT pushed) — the oldest T0 metrics debt fully RETIRED:** `/metrics` now serves LIVE values, not the primed schema. `EngineCoreOutputs.scheduler_stats` (new `Scheduler::make_stats()`, `scheduler.py:2399-2436`) + a stamped `timestamp`; `OutputProcessor::process_outputs` builds `IterationStats` (token counts, TTFT/ITL, finished-request breakdowns off `RequestState` timing, `stats.py:377-475`); the sync `LLMEngine::step()` folds both into the attached logger's `Record()` (`llm_engine.py:308-329`). Behavioural CPU gate `test_llm_engine.cpp` case 6 (44 assertions) drives the reference engine several steps: running/waiting gauges track the batch, prompt/generation counters == exact token counts, `request_success_total` counts finished reqs, TTFT/ITL/e2e/TPOT/iteration histograms observe the right sample counts; RED-first (14 asserts flip 0→correct with `Record` off). Additive/opt-in: null logger ⇒ byte-identical greedy stream; catalog gate `test_prometheus_metrics` 4/4/81 unchanged. **PER-REQUEST TIMING via EngineCoreEvents LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-RESPONSE-METRICS`, NOT pushed) — `SERVE-RESPONSE-METRICS` INVENTORIED→ACTIVE:** the scheduler records QUEUED/SCHEDULED/PREEMPTED `EngineCoreEvent`s 1:1 vLLM (`scheduler.py:2135/1003/1221`, gated on `log_stats_`), drained onto `EngineCoreOutput.events`; `OutputProcessor.update_from_events` folds them into `FinishedRequestStats.{queued,prefill,inference}_time` + `IterationStats.num_preempted_reqs`, so `vllm:request_{queue,prefill,inference}_time_seconds` + `vllm:num_preemptions_total` now carry real durations (the live-metrics wiring left them at 0). Gates: `test_scheduler` +1 (15 asserts, RED-first, real KV-exhaustion preemption) + `test_llm_engine` +1 (26 asserts, RED-first: 5 flip 0→positive; inference=prefill+decode, prefill≤inference≤e2e); no-logger path byte-identical. **CHAT-FORM `/tokenize` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-CHAT-TOKENIZE`, NOT pushed):** `/tokenize` accepts BOTH arms of the `TokenizeRequest` union — the chat form (`messages`+flags) renders through the SAME model chat template `create_chat_completion` uses (`chat_.prompt_fn()`), applies `check_generation_prompt` (→400), tokenizes with the chat-form `add_special_tokens` default False, returns `{count,max_model_len,tokens,token_strs?}` identically to vLLM 0.26; exact-gated tokens == render→Encode, RED-first (`test_openai_api_server` 27/27/337); raw-`prompt` form byte-identical. **JSON-SCHEMA ARG-TYPE COERCION LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-ARG-COERCION`, NOT pushed):** `_fix_arg_types`/`_streamable_string_keys`/`find_tool_properties` (parser_engine.py:227,269,365,348) ported over the already-ported `extract_types_from_schema`/`coerce_to_schema_type` helpers — `ParserTool` carries the function `parameters` schema (threaded from `serving_chat.cpp` `ToParserRequest`), so a request whose tools declare typed params has its assembled `tool_calls[].function.arguments` coerced to int/number/bool/string/array/null in BOTH streaming (`parse_delta`) and one-shot (`extract_tool_calls`/`parse`), 1:1 with vLLM (priority null>int>number>bool>object>array>string, uncoercible values left as-is); no-schema/absent-tools = identity byte-identical. Gate `test_parser_engine_assembly` 30 scenarios, 5038/5038 (scenarios 27-30: qwen3 typed-schema whole+char, schema-mismatch, kimi JSON-native `"5"`->int), RED-first 38 asserts (`"5"`-vs-`5`); engine-core 586/586 + serving-SSE 210/210 byte-identical. **`/tokenizer_info` (`SERVE-UTILITY-ENDPOINTS`) + `/abort_requests` (`SERVE-ADMIN`, INVENTORIED→ACTIVE) LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `GET /tokenizer_info` gated behind a `set_tokenizer_info_enabled` flag mirroring vLLM's `enable_tokenizer_info_endpoint` (off→route absent→404) surfaces the `tokenizer_config.json`-equivalent fields our byte-level/SentencePiece BPE tokenizer can GENUINELY back (`tokenizer_class`/`model_max_length`/`vocab_size`/`bos`/`eos`/`added_tokens_decoder`) and OMITS (never fabricates) the named gaps — the raw `chat_template` string (lives in the ChatPromptFn seam), the HF `init_kwargs`, and added-token `normalized`/`single_word`; `POST /abort_requests` (dev/rlhf admin router) parses `{request_ids}` and aborts the listed ids via an injected engine-abort callback (`AsyncLLM::abort`), empty→"abort all", returning `{"status":"aborted","aborted":N}` (malformed→400 `{"detail":...}`, failure→500 `{"error":...}`) — all shapes 1:1 vLLM. Both additive/opt-in (route only when backing attached), RED-first over a real socket (404 off → 200 on) + behaviour (abort tears down an in-flight AsyncLLM request); `test_openai_api_server` 31/31/379, existing conformance 23/23 + serving 40/40 byte-identical. **PRODUCTION `main.cpp` ENDPOINT WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary previously called NONE of the opt-in endpoint setters — a whole family of unit-tested endpoints was DARK in the actual server. A single shared seam `ConfigureUtilityEndpoints` (exercised by both `main.cpp` and the gate over a real socket) now lights, mirroring vLLM 0.26's per-endpoint default gating: `/tokenize`+`/detokenize` (on by default when a tokenizer exists), `/tokenizer_info` (new `--enable-tokenizer-info-endpoint`, mirrors `cli_args.py:140`), and `/abort_requests` (new `--enable-server-dev-mode`, mirrors `envs.VLLM_SERVER_DEV_MODE` at `api_server.py:238`) wired to the LIVE `AsyncLLM` with an exact before/after abort delta-count. `/metrics` + `/reset_prefix_cache` stay NAMED-BLOCKED (the AsyncLLM frontend exposes no live `PrometheusStatLogger` and no thread-safe prefix-cache reset RPC — attaching either would be a fabricated wiring). Core routes byte-identical; `test_openai_api_server` 32/32 (420, RED-first no-seam→404), conformance 23/23 + serving 40/40 unchanged; clean CPU `-Werror` incl. the `server` binary. RESIDUAL (honest): the chat/completion RESPONSE-BODY timing surface + CLI validation; AsyncLLM production-serving metric wiring (`/metrics` backing); config-gated metric families (spec-decode/kv-connector/mm/LoRA); `chat_template_kwargs`/`continue_final_message` full render passthrough; `/abort_requests` empty-list "abort ALL" enumeration; live `/reset_prefix_cache` backing | `PARTIAL` | response-body timing; AsyncLLM serving-path metric wiring + config-gated families; then DONE | | 9 | `ROAD-V1-C9` | Mechanical recurring upstream sync | [upstream sync](upstream-sync.md), [v0.25 audit](sync/2026-07-12-702f481.md), [porting inventory](porting-inventory.md) | v0.25.0 target `702f481` enumerated/classified: 145 post-pin commits, 94 inventory and 51 ignore, no unequivalent PORT-NOW runtime delta in the implemented T0 slice. The executable DGX oracle is validated/active at the target; the porting pin remains `e24d1b24` pending target goldens/behavior/model re-verification | `PARTIAL` | refresh exact performance denominators and target goldens/tests, then advance the parity pin | -| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **RPi5/A76 R4-R5 GREEN; `KERNEL-CPU-A76-Q8-DOT` GATING:** QEMU-built artifact is token/fixture exact on Pi; scheduled AAPCS64 beats compiler SDOT 3.66-5.08% on M1/T1 and M128, lowers 64-token Qwen TTFT 1.55% and E2E 0.13%. A76+DotProd auto-selects assembly; M1/T4 (−2.43%), memory/concurrency and same-file llama.cpp floor remain open. **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | +| 10 | `ROAD-V1-D1` | NVIDIA target fan-out, ROCm, MLX, Vulkan, XPU, ANE | [backend matrix](backend-matrix.md), [backends strategy](backends.md), [arch additivity](specs/cuda-arch-additivity.md) | **RPi5/A76 R4-R5 GREEN; `KERNEL-CPU-A76-Q8-DOT` GATING:** QEMU-built artifact and same-prompt model output are exact; scheduled AAPCS64 beats compiler SDOT 3.66-5.08% on M1/T1 and M128. The same-file llama.cpp gate is now MEASURED/NOT MET on speed: vllm.cpp is 0.461x prefill and 0.653x decode/E2E, but uses 24.2% less peak RSS. M1/T4 (-2.43%), BF16 GEMM/speed closure and concurrency remain open. **The CUDA arch-expansion SEAMS are landed and gated (`BACKEND-CUDA-ARCH-ADDITIVITY`, `PARTIAL`):** the four hardcoded `12[01]a` CMake guards became a per-arch FEATURE TABLE (1:1 port of vLLM's `cuda_archs_loose_intersection`), the device capability is cached and threaded to the kernel layer and carried by the backend, host launchers select a tactic at runtime from a registration point, and the opt-in shared-memory ceiling is queried instead of assumed. Adding an arch is now a table-row edit + a tactic registration. This is MECHANISM ONLY — exactly ONE tactic is registered (the existing `sm_12x` fp4 path), behavior on GB10 is unchanged; per-source gencode narrowing and every cross-family tactic body stay HW-blocked. **THE MECHANISM IS NOW EXERCISED: `BACKEND-CUDA-SM120` (consumer Blackwell, RTX 50-series) is BUILD-SUPPORTED as of 2026-07-22 (`CLAIM-CUDA-SM120-BRINGUP`, [spec §W8](specs/cuda-arch-additivity.md)) — the FIRST arch to move past `INVENTORIED`, and it needed ZERO kernel, model, runner or feature-table edits, confirming the audit's §A.3 same-family prediction by measurement.** The known heterogeneous-fat-build obstacle turned out to be FAMILY-SCOPED: `"120a;121a"` configures, compiles `-Werror` 0-warn and produces real `sm_120a` SASS in 38 TUs (`cuobjdump -lelf`), because the sm12x-only PTX and the `12[01]a`-shaped guards cover both; `"90a;121a"` still cannot compile and W7 still owns that. Triton AOT stays SINGLE-ARCH by the nature of cubins (a vendored artifact will not `cuModuleLoadData` on another SM), so a fat build needs `-DVLLM_CPP_TRITON=OFF` — which degrades gracefully onto the portable CUDA kernels — and the derivation now DIAGNOSES the multi-arch case instead of failing on a nonexistent joined directory name; **no cubins were faked for an architecture we cannot run.** **This is a BUILD claim, NOT runtime support:** no sm_120 board exists here, no gate model has executed on one, and the row stays `PARTIAL`. **THE CROSS-FAMILY FAN-OUT IS NOW BUILD-SUPPORTED TOO (2026-07-26, `CLAIM-CUDA-ARCH-EXPANSION`, [spec §W10](specs/cuda-arch-additivity.md)):** after sm_90a (Hopper, W9), the remaining cleanly-additive arches vLLM builds for — Ampere/Jetson/Ada `sm_80/86/87/89`, datacenter Blackwell `sm_100a/103a`, `sm_110` — moved `INVENTORIED` -> BUILD-supported, PORTABLE-KERNELS-ONLY, mirroring sm_90a (all five fast-path FEATURE-TABLE cells resolve EMPTY). The per-major representatives `sm_80`/`sm_100a`/`sm_110` compiled `-Werror` 0-warn on dgx with 16 TUs of real per-arch SASS each; **ZERO kernel/model/runner edits** (W9's guards generalized the compile), so the sm_121a default is byte-identical. `sm_70`/`sm_75` (no bf16 tensor cores) and `sm_101a` (not in nvcc 13.0) are recorded SCOPED, non-additive — a real fp16 kernel body, not a table edit. Still a BUILD claim only: no such board here, no execution, and NO fast-path kernels for any of these families (a vLLM-competitive path is the per-family wgmma/tcgen05/Ampere kernel campaign). 13 CUDA targets, component rules, platforms and native floors inventoried. SGLang v0.5.13 preflight P1 remains implemented/`GATING` with 16 CPU tests; image/model P2 and cache-neutral binding evidence remain open. A distinct v0.5.15 shared-prefix row is now fully spiked/`READY`, with the external scalar rejected and PX1/PX2 exact long-prefix harness plus Mamba-align retention next **NON-CUDA FAN-OUT NOW SPIKED (2026-07-22, `CLAIM-BACKEND-FANOUT-1`, [backend fan-out](specs/backend-fanout-metal-vulkan-xpu.md)):** USER PRIORITY 3 was genuinely untouched; `BACKEND-METAL-MLX`, `BACKEND-VULKAN` and `BACKEND-XPU` move `INVENTORIED` -> `SPIKE` on MEASURED hardware verdicts, not inferred ones. **Metal is REAL and gateable TODAY** — the tree configures and builds on the M4 under CLT-only AppleClang with three Clang-only `-Werror` fixes, and **108,952 portable-tier assertions already pass on Apple arm64** (a third-architecture confirmation of the GGUF quant tier); the sole blocker was `CMakeLists.txt:304-306` silently dropping every static registrar on Apple (`no backend registered for device type 0`), and the one-line `-force_load` fix was applied and verified GREEN 7/7. Runtime MSL compilation works with NO Xcode and NO MLX, so bring-up is re-sequenced to native MSL (E2) with MLX (E1) demoted to a later measured comparison. **Vulkan is REAL and uniquely gateable** — GB10 enumerates as `INTEGRATED_GPU` at API 1.4.312 with `VK_KHR_cooperative_matrix` v2 AND `VK_NV_cooperative_matrix2`, on the same box that runs our CUDA backend as the oracle, plus a software ICD on the dev box for GPU-free CI; llama.cpp `ggml/src/ggml-vulkan/` is locally readable at our own pin as the port source. **XPU is HW-BLOCKED and doubly so** — no Intel GPU anywhere, and vLLM has NO in-tree SYCL source to mirror (its XPU kernels are the external `vllm_xpu_kernels` package), so only policy/compile/CPU-device unit work is proposed. Ranked Metal > Vulkan > XPU. NO implementation, NO kernels, NO benchmark; `BACKEND-ROCM`/`BACKEND-ANE` and every native-competitor gate row stay `INVENTORIED`. **METAL W0 LANDED 2026-07-22 — the non-CUDA half is no longer spike-only.** The shared seam repair plus a Metal `vt::Backend`/`Platform` SKELETON are merged and gated on all three boxes. The headline fix is bigger than Metal: `CMakeLists.txt` gated the static-registrar force-link behind `if(UNIX AND NOT APPLE)`, which is FALSE on Apple, so ld64 dropped EVERY registrar and even the **CPU backend** failed to register on macOS (`test_backend` 5/7 FAIL); with the `-force_load` branch it is 7/7 and the whole tree builds `-Werror`-clean on the M4 — so `BACKEND-CPU` gains macOS as a usable target too. The spike predicted three Clang-only diagnostics; a FULL build surfaced **seven**, and **every one was FIXED, none suppressed** — including a genuine `-Wdelete-non-abstract-non-virtual-dtor` UB (`AsyncScheduler` owned through `unique_ptr` with a non-virtual base dtor), which is a latent defect on EVERY platform, not a macOS annoyance. The skeleton compiles MSL AT RUNTIME via `newLibraryWithSource:` (no Xcode, no MLX, zero installs), pins `MTLMathModeSafe` so IEEE semantics survive, and registers 8 ops plus ONE `kFusedChain` interpreter that inherits the whole portable fusion catalog. It also closes the spike's named "harness gap" with `tests/vt/test_backend_cross_device.cpp`, a CPU-oracle equality harness that serves EVERY registered non-CPU backend (Metal today, Vulkan/XPU unchanged later) — and which immediately earned its keep by catching two skeleton bugs and one real CPU-vs-CUDA NaN-payload difference. Op NMSE vs the CPU oracle worst-case **1.9e-14 against the 5e-4 bar**, with Copy/Memset and the bf16<->f32 codec bit-exact. **`BACKEND-METAL-MLX` moves `SPIKE` -> `ACTIVE`, which means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are all unregistered and NO model runs on Metal.** All six dgx regressions re-verified UNCHANGED. Vulkan and XPU stay `SPIKE`. **NEW BINDING CONSTRAINT (user directive 2026-07-22): MLX is the named COMPETITOR FLOOR for Metal** (`BACKEND-GATE-METAL-MLXLM`) — same model, same workload, match-or-beat on every axis — which is INDEPENDENT of its demotion as an implementation path, and which makes **Qwen3-dense** the first Metal bring-up model (MLX-LM runs it, and it satisfies the spike's "never Qwen3.5-Next" constraint). `PARTIAL` **VULKAN V1 LANDED 2026-07-22 — the non-CUDA half now has TWO gated skeletons.** The Vulkan `vt::Backend`/`Platform` skeleton is merged with the SAME eight ops plus the `kFusedChain` interpreter as Metal, so the two are directly comparable through one harness. Two things make it structurally more valuable than Metal despite arriving second. **(1) The oracle is our own CUDA backend, in the same binary, on the same box** — `test_backend_cross_device` runs **144 assertions on GB10 versus 73 on a Vulkan-only box** precisely because it compares the CPU reference against BOTH CUDA and Vulkan in one process; that is the strongest cross-backend check anywhere in this project and Metal cannot have it. **(2) The build needs no shader toolchain at all.** llama.cpp shells out to `glslc` at build time; neither of our boxes HAS a GLSL compiler, has the Vulkan dev package, or grants sudo, and linking libshaderc would be a forbidden compiled dependency — so the GLSL is compiled AHEAD OF TIME (glslang 16.4.0) and the SPIR-V is COMMITTED, which also permanently side-steps the spike's Risk 4 (Ubuntu's shaderc 2023.8 silently failing the coopmat2 probe). Entry points are `dlopen`ed rather than linked, so the library still loads on a box with no Vulkan; the Khronos headers are vendored as TYPES ONLY under `VK_NO_PROTOTYPES`. The relaxed-precision traps were found and handled: `1.0/sqrt` replaces llama.cpp's `inversesqrt` (the Vulkan analogue of Metal's `MTLMathModeSafe` pin), no `RelaxedPrecision` is emitted, and the fp32 float-controls that GLSL cannot pin are PROBED and reported instead of assumed. **It also runs on `llvmpipe` with no GPU present — the GPU-free CI path is now PROVEN, not projected.** `VLLM_CPP_VULKAN` AUTO resolves **OFF** (deliberately unlike Metal's AUTO) so the CUDA gate builds are untouched, and all six dgx regressions were re-verified UNCHANGED to prove it. **`BACKEND-VULKAN` moves `SPIKE` -> `ACTIVE`, which again means a GATED SKELETON, NOT a supported backend: GEMM, attention, KV cache, quant and sampling are unregistered and NO model runs on Vulkan.** XPU stays `SPIKE` and hardware-blocked. **METAL/MLX REUSE STUDY + THE `vt::OpProvider` SEAM + AN MLX BASELINE LANDED 2026-07-22 ([study](specs/metal-mlx-reuse-study.md)), answering the user's extensibility directive directly.** Docs+measurement only — no source, CMake, kernel or test changed. **(1) The reuse is HIGH and now exact:** a first model on Apple GPU needs **10 ops for Qwen3-dense (7 new)** or **9 for OPT (6 new)**; the engine, every `ops.cpp` argument check (a backend writes ZERO validation), the whole 9-recipe fusion catalog, the sampler source and weight loading (**0 OpIds**) all transfer unchanged, and **OPT's four TUs contain ZERO CUDA references**, making it the cheapest correct first non-CUDA model. **(2) The seam verdict:** a new PLATFORM plugs in through what exists — MEASURED, since Vulkan V1 edited exactly two pre-existing files and the cross-device harness picked it up unmodified — but a second PROVIDER on an existing platform CANNOT, because `src/vt/ops.cpp:10-15,98-102` is a flat `[OpId][DeviceType]` table of one `void*` with **silent last-writer-wins** under unspecified static-init order. We already built the right mechanism one layer down: `cuda_arch_tactics` has capacity-bounded static storage, capability predicates, first-match selection with a portable fallback, a decline-by-return second axis, and selection stats — and its CUDA couplings are **exactly 3 and all shallow**, verified. **Recommendation: generalize it INTO the op table as `vt::OpProvider` — ~50 lines, ZERO call-site edits, and ONE mechanism serving MLX-on-Metal, cuBLASLt/CUTLASS/flashinfer-on-CUDA and llama.cpp-on-CPU/Vulkan.** This merges W0b item 8 (they are the same work) and restates item 7 (`QuantTypeTraits` must be SPLIT, not lifted — `vec_dot_type` and `nrows` are implementation facts, not encoding facts, and MLX's own `(group_size, bits, mode)` convention proves the axis is real). **(3) MLX integration is IN SCOPE and the lazy-eval objection is REFUTED BY SOURCE** — `backends.md:84-90` was wrong: MLX's graph terminates at `mlx/backend/metal/eval.cpp:32-48` calling `primitive().eval_gpu()`, an eager per-op encode, and the compute entry points are FREE FUNCTIONS (`steel_matmul`, `matmul.h:105-142`); a zero-copy bridge exists because `allocator::Buffer` is a bare `void*` that IS the `MTL::Buffer*`, `array::set_data` takes a caller-supplied deleter, and `set_input_array`'s offset arithmetic is already correct on our `StorageModeShared` buffers. So MLX becomes an **optional gated provider** (`VLLM_CPP_MLX`, default OFF — precedent `VLLM_CPP_TRITON`), a CONFIGURATION rather than a rewrite; native MSL stays the default because it needs zero installs and because **MLX has NO paged-KV attention**, so `kPagedAttention`/`kReshapeAndCache` are ours regardless. Honest cost measured: `libmlx.dylib` + a **104,894,650-byte `mlx.metallib`**. **(4) MLX BASELINE MEASURED — `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** `mlx` 0.29.3 / `mlx-lm` 0.29.1 installed via the venv route (brew NOT used, so `python@3.14` never touched the build PATH), `mlx-community/Qwen3-1.7B-bf16`, MLX-LM's OWN harness — **27.57 tok/s @ b=1 rising to 213.39 @ b=16**, TTFT 470 ms -> 6,857 ms, peak 3.78 -> 5.28 GB, trial spread 0.12-0.63%. The `com.localai.worker` daemon could not be stopped (no passwordless sudo) though it was MEASURED genuinely idle (0.0% CPU, `ioreg` GPU utilization 0); a second unanticipated contender — the desktop aerial video wallpaper at 8.2% CPU — was found and recorded. **NO "ours" column exists or was manufactured and NO Metal speed result is claimed.** **`BACKEND-GATE-METAL-MLXLM` deliberately STAYS `INVENTORIED`** — a competitor baseline is not a gate, and with no implementation code and no ours-vs-MLX evidence there is nothing to anchor; it moves at work row `M3b`. **(5) Two open defects found by inspection, neither fixed:** `dense_attn_block.h:140,157` hands a HOST pointer to a DEVICE kernel on any non-CUDA device backend (a real bug, latent only because no model runs there — hard M3 blocker), and `test_fused_chain_additivity.cpp:420-439` gates 7 of the 9 declared recipes so its count guard has drifted. **THE PROVIDER SEAM IS NOW LANDED AND GATED (`BACKEND-ACCEL-PROVIDER`, `ACTIVE`, 2026-07-22)** — the piece this whole row needed and did not have. Adding a new PLATFORM was already additive (Vulkan V1 edited exactly two pre-existing files); adding a second PROVIDER within a platform was IMPOSSIBLE, because the op table held one `void*` per (OpId, DeviceType) and last-writer-wins under unspecified static-init order made "two implementations of one op" a nondeterministic build rather than a configuration. `vt::OpProvider` replaces it with deterministic `(priority, name)` selection, a device-neutral capability predicate, per-call decline-and-fall-back, and selection stats — ONE mechanism intended to serve MLX-on-Metal (object-model C++), cuBLASLt/CUTLASS/flashinfer-on-CUDA (raw C launchers, whose ARGUMENT half is `dropin-kernel-abi.md`), llama.cpp on CPU/Vulkan, and the existing CUDA arch tactics it was generalized from. **First provider proven end-to-end: MLX for the Metal dense GEMM**, alongside a new native MSL GEMM which stays the default. Still HW-blocked/unpopulated: the CUDA, CPU and Vulkan provider rows, and no model runs on any non-CUDA backend. **=== 2026-07-22: A MODEL NOW RUNS ON A NON-CUDA BACKEND, AND IT RUNS TOKEN-EXACT (work row `M3a`) ===** OPT-125m (`OPTForCausalLM`) generates end to end on Apple GPU and is **STRICT token-exact 6/6 prompts / 96/96 tokens** vs the SAME committed dgx-captured vLLM 0.25.0 goldens the CUDA arm is gated on — a device-INDEPENDENT bar, so Metal met the bar CUDA already met rather than one re-derived on Metal. Five new MSL kernels took Metal to **15 of 75** ops (`kEmbedding`, `kQkvSplit`, `kReshapeAndCache`, `kPagedAttention`, `kGreedyArgmax`); four are BIT-EXACT vs the CPU oracle and `kPagedAttention` is NMSE 4.99e-13 against the 5e-4 bar (bit-exactness explicitly NOT claimed for it — online vs materialized softmax is a different reduction order by construction). The Metal path is PROVEN to have executed (device type asserted, `selections > 0` and `declines == 0` on all nine ops, NaN-poisoned outputs), not inferred. Closing it required 2 of the study's predicted 4 seam fixes, REFUTED a third by measurement, and surfaced **one bug neither the spike nor the study predicted** — `runner.cpp:516` gated KV-cache device residency on `is_cuda()`, handing a HOST pointer to a device kernel on any non-NVIDIA accelerator — plus forced a new `Platform::supports_model_architecture()` seam, because once queue selection asks the platform, "which device am I on" stops being "which device can run THIS model". **NO Metal SPEED number is claimed or owed** (the M4 could not be quieted), so this is a CORRECTNESS milestone only and the row stays `PARTIAL`. **=== 2026-07-23: A SECOND MODEL RUNS ON A NON-CUDA BACKEND + THE FIRST OURS-VS-MLX BENCHMARK (work row `M3b`, `CLAIM-BACKEND-METAL-M3B-1`) ===** Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B) generates end to end on Apple GPU; the forward is CONFIRMED CORRECT by the vLLM oracle (near-tie-robust, NOT strict-token-exact — 0.6B is a near-tie model). The Metal forward resolves the model's genuine near-ties the other way (p0 tok5 15344 " Italy" vs 9625 " France"); the DECISIVE oracle measurement (`scripts/qwen3-neartie-gap.py` teacher-forcing vLLM 0.25.0 on the METAL prefix, batch=1, `gpu_mem_util=0.40`) shows all 60 Metal-vs-CUDA divergent positions within **0.5 nats of vLLM's own argmax given the Metal prefix — max 0.125 nats, none outside top-20** (p0 tok5 gap 0.0000 — vLLM's teacher-forced argmax there IS Italy, contradicting its CUDA-capture France pick). The gate PASSES **16/16** against Metal's OWN oracle golden (`our_ids_metal.npy` + `neartie_gap_mnats_metal.npy`; 10 strict token-exact vs vLLM greedy + 6 near-tie-band, 0 forward-divergent), identical gate logic to CUDA, teeth proven (perturbed anchor/gap both FAIL). Metal execution proven (device==kMETAL, all 9 ops `selections>0 ∧ declines==0`). STRICT token-exactness on 0.6B is ill-posed; a strict Metal gate wants a bigger deterministic dense model (Qwen3-4B, not on the M4) — DEFERRED. Three new MSL RoPE kernels took Metal to **18 of 75** — the study's op prediction corrected: the DEFAULT `VT_QWEN3_ROPE_CACHE`-ON path needs `kRopeCosSinCache` (build the per-step cache) + `kRopeFromCache` (apply, bit-exact), `kRopeNeox` the cache-off opt-out. **FIRST OURS-VS-MLX BENCHMARK (Qwen3-1.7B, same box/session, INDICATIVE/BLOCKED-ON-SUDO):** ours (Metal) is a knowingly-unoptimised FLOOR — ~6–11× slower decode, ~7–10× slower TTFT, ~2× peak memory vs MLX's steel kernels — NOT binding (Mac could not be quieted); `BACKEND-GATE-METAL-MLXLM` moves `INVENTORIED` -> `ACTIVE` with the exact residual (close via `M3c` batched encoders + a simdgroup GEMM, then re-run quiet). The dgx CUDA Qwen3-dense gate is GREEN (the earlier "currently red / stale golden" claim is DISPROVEN): the France/Italy tie is BUILD-SENSITIVE — the PRODUCTION dgx build (FA2+Marlin+Triton+CUTLASS) resolves p0 tok5 → France 9625 and passes 16/16 (0.6B: 10 strict + 6 near-tie, max gap 0; Qwen3-4B: 16/16, max 0.25 nats), while a portable-kernel-only CUDA build resolves it → Italy 15344 (matching Metal). The golden is the production build's resolution of a genuine numerical near-tie, NOT stale. Regressions GREEN (OPT 6/6, Qwen3-Coder 6/6; 27B/35B/DeepSeek byte-identical); goldens md5 `2965ef5772b556d3f3f86fedf4221b2f` unchanged. | `PARTIAL` | after the active 27B cache-off closure, repin/provision the digest-pinned v0.5.15 image, resolve raw E2E/TPOT detail without changing timed semantics, classify exact checkpoints, and execute the shared-prefix gate only after hit/dtype/capacity equivalence **PARALLEL TRACK OPENED:** the CUDA-arch half stays HW-blocked, but the non-CUDA half is now actionable — start `W0` (the 8 shared seam repairs, behavior-preserving on Linux/CUDA by construction) then `M1`/`M2` Metal bring-up on the M4, gated against our OWN CPU backend since vLLM cannot run there; Vulkan `V1` follows, gated against our OWN CUDA backend on GB10. **NEXT ON THE NON-CUDA TRACK:** `M2` residue (the ~12 remaining elementwise/rope/gated ops) then `M3` (GEMM + paged attention -> the first Metal model, **Qwen3-dense**, token-exact vs our own CPU backend AND benchmarked against MLX-LM on the same M4). `W0b` (spike port-map items 5-8: guard the 4 `vt/cuda/` includes incl. the PUBLIC `dense_nvfp4_gemm.h:66`, de-hardcode the `kCUDA` loader queue, lift `QuantTypeTraits` out of `vt::cpu`, generalize the arch-tactic registry) is deferred but is a PREREQUISITE for M3 — those four are exactly what blocks a model on a non-CUDA backend. Install MLX on the M4 via the venv route and boot out the `com.localai.worker` daemon before any timing. Vulkan `V1` follows, gated against our own CUDA backend on GB10 using the same cross-device harness. **SEAM AUDIT LANDED 2026-07-22 (`BACKEND-SEAM-AUDIT`, now `ACTIVE` — audit+plan, plus work row `S1` implemented; still no build and no GPU):** the user's direct question *"are MLX/Vulkan porting the same strategy as vLLM's CUDA path, and does it map correctly?"* is answered **PARTIAL** in [accelerator-seam-audit.md](specs/accelerator-seam-audit.md). MIRROR at the two seams we ported (`platforms/interface.py:134-229`, `v1/attention/backends/registry.py`) — and our attention self-registration is OPEN where upstream's enum is CLOSED, a divergence in our favour. **ABSENT at `model_executor/layers/`**: no `QuantizationConfig`/`LinearMethod`, so the device branching upstream absorbs in its shared layer library lands in our model TUs instead — shared-layer **DSR = 94** at `72f5db2`, **67 (71%) of it in `qwen3_5.cpp`**, versus **1** predicate in its 802-line upstream twin `qwen3_next.py:321` (upstream: 199 of 544 predicates in `layers/`, only 14 across 287 model files). Granularity verdict: our 75-op `vt::` table is NOT finer than upstream's 42 `CustomOp`s; the real asymmetry is `CustomOp.forward_native` giving vLLM a ZERO-op correctness floor where `vt::GetOp` throws. Ranked plan `S1`-`S8` (DSR CI ratchet -> the owed `W0b-1` residue -> platform capability fields -> portable reference tier -> `LinearMethod` -> fast-path gates -> layer-library extraction -> XPU data-only), each needing its own claim. **`S1` LANDED 2026-07-22 (`CLAIM-BACKEND-SEAM-S1-1`; row now `ACTIVE`): the DSR ratchet is CI-gated** — `scripts/check-device-leakage.py` + a committed baseline + the `device-leakage` CI job + a 24/24 mutation suite, all standalone with no CUDA toolkit and no GPU, and **not one file under `src/`/`include/` touched**. **Baseline re-derived from the current tree as required, and it MOVED: 94 -> 86** at `18094ee` (44 `kCUDA` + 10 `is_cuda()` + **0** unconditional CUDA includes + 32 `#ifdef VT_*`) — **-6 is a composition correction to the audit itself** (comments/strings are now stripped uniformly, and its 4 "unconditional" CUDA includes were ALL already `#ifdef`-guarded, a measurement error) and **-2 is real reduction** that landed on main since `72f5db2`. Re-verifying the seven-class split site-by-site SHARPENS the audit's verdict: only **3 of 86** are genuine device policy (was 6), 46 are fast-path availability the op table already answers, 5 duplicate `GetOp`'s own throw, 32 are build config. Leakage is now MEASURED AND FENCED, not merely described. **`S4` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S4-1`): the `LinearMethod`/`QuantizationConfig` coarse seam** (`linear.py:141-230` + `base_config.py:20-229`) is built and the dense model routed through it; 18 provably-`kCUDA`-only device gates became `vt::OpRegistered` — **DSR 86 → 67**, byte-identical (27B 235/235 · 35B 315/315 · Coder 6/6 · dense 16/16 · OPT 6/6 · DeepSeek-V2 8/8), the fragile fp4-act/fp8-recipe gates deferred to `S6`. **`S5` LANDED 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`): the portable reference tier** — the literal answer to the "correct with zero kernels" cliff. Mirroring `custom_op.py:138 forward_native`, a CPU kernel installs LAZILY as a negative-priority (`-1000`) `vt-cpu-ref` `OpProvider` on a UNIFIED-MEMORY device's first `GetOp` miss, so a partial backend (Metal 18/75 ops, Vulkan skeleton) RUNS an op it lacks natively instead of throwing. Op count became a PERFORMANCE budget, not a correctness gate. SAFETY = gated on `Backend::UnifiedMemory()`, never DeviceType (a discrete GPU never gets a CPU fallback — proven against a fake discrete backend); CUDA byte-identical (native always wins by priority + install is lazy-on-miss, and CUDA never misses on the gate models); observable via `GetReferenceTierHits()`. **DSR unchanged at 67** — S5 is vt-runtime infrastructure, no shared-layer edit. It UNBLOCKS `S6` (it is `S6`'s safety net); the S4-deferred fp4/fp8 gates stay deferred (their ops are dual/CPU-registered, so `OpRegistered` ≠ `==kCUDA` on a CPU device — not byte-identical, verified). `S2`,`S3`,`S6`-`S8` remain unclaimed and no other implementation row moved. | | 10a | `ROAD-V1-D1-CUDA` | **CUDA-arch derive-and-ship + community testing matrix (user-directed 2026-07-27):** the arch-additivity framework already makes a new CUDA device a near-verbatim add, so DERIVE-AND-SHIP the fast-path kernels rather than scope-and-wait — port 1:1 from vLLM (Ampere FA2/Marlin/AllSpark/scaled-mm-C2x; Hopper FA3/C3x-Sm90/Machete/MoE; datacenter-Blackwell C3x-Sm100a-tcgen05/MLA/fp4), and the fp16/non-tensor-core bodies from llama.cpp for the arches vLLM DROPS (Pascal/Volta/Turing = MORE than vLLM), build-verify (compile + `cuobjdump` SASS), and SHIP LABELED with a 3-state signal: RUNTIME-VERIFIED / DERIVED+BUILD-VERIFIED (testing-welcome) / NOT-YET-BUILDABLE (a build is never a runtime claim). Runtime-gate boards: **AGX Orin (sm_87, reachable)** + **NVIDIA Thor (Blackwell, reachable, SM to confirm on-box)** — the first two non-GB10 runtime proofs, spanning Ampere + Blackwell. DeepGEMM (runtime JIT) is the one honest NOT-YET-BUILDABLE. | [backend matrix](backend-matrix.md), [Ampere](specs/cuda-arch-ampere-fastpath.md), [Hopper+DC](specs/cuda-arch-datacenter-fastpath.md), [breadth](specs/cuda-arch-breadth-fp16.md) | 3 committed spikes (2026-07-27); backend rows moved INVENTORIED/build-only -> SPIKE; **FIRST NON-GB10 RUNTIME PROOF: Thor sm_110 portable bf16 path RUNTIME-VERIFIED (token-exact Llama-3.2-1B, `CLAIM-CUDA-SM110-RUNTIME`)**; **FIRST DERIVE-AND-SHIP BRICK: WA-1 FA2 Ampere enablement DERIVED+BUILD-VERIFIED (fa2 cell widened to 8.x; sm_87/sm_80 FA2 cubins, sm_121a SACRED 16/16 unchanged, no Ampere board ran it)**; **Turing sm_75 W1 BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-TURING-SM75`): the bf16-WMMA TU guard landed, single-arch `75` `-Werror` 0-warn + `cuobjdump` real `sm_75` cubin, sm_121a byte-identical — DERIVED+BUILD-VERIFIED (testing-welcome), no board ran it**; **DC1 datacenter-Blackwell sm_100a NVFP4 tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-NVFP4`): faithful 1:1 port of vLLM `Fp4GemmSm100` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05) as a NEW dedicated `cutlass-nvfp4-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror`-equiv 0-warn + `cuobjdump` real `sm_100a` cubin, RED (HEAD 100a fp4 DISABLED), sm_121a neutral — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; native `fp4-mma` stays sm_12x-only (mma.sync≠tcgen05)**; **DC2 Hopper sm_90a CUTLASS C3x FP8 scaled-mm BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM90-C3X`): faithful 1:1 port of vLLM `cutlass_3x_gemm_sm90_fp8` (`ArchTag=Sm90`+`KernelTmaWarpSpecialized*FP8FastAccum`→wgmma/TMA) as a NEW dedicated `scaledmm-c3x-sm90` FEATURE-TABLE cell (90a-only), single-arch `90a` `-Werror` 0-warn + `cuobjdump` real `sm_90a` cubin (ptxas C7510 names `wgmma.mma_async`), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no Hopper board ran it; sm90 int8/blockwise + sm100 C3x legs are residual bricks**; **DC3 datacenter-Blackwell sm_100a CUTLASS C3x FP8 scaled-mm tcgen05 GEMM BUILD-VERIFIED 2026-07-28 (`CLAIM-CUDA-SM100-C3X`): the intersection of DC1's tcgen05 arch + DC2's C3x fp8 kernel — a faithful 1:1 port of vLLM `cutlass_3x_gemm_sm100_fp8` (`ArchTag=Sm100`+`KernelScheduleAuto`→tcgen05, `sm100_fp8_config_{default,M256,M64}`, 2SM `<_2,_2,_1>` default) as a NEW dedicated `scaledmm-c3x-sm100` FEATURE-TABLE cell (100a-only), single-arch `100a` `-Werror all-warnings` 0-warn EXIT=0 + `cuobjdump` real `sm_100a` cubin (`Sm100TmaUmma`/`SM100_MMA_F8F6F4_2x1SM`/`TMEM` symbols, `LDTM`/`tmem` SASS), RED (HEAD had no cell), sm_121a neutral (`cutlass-fp8`/`cutlass-nvfp4`/`cutlass-nvfp4-sm100` unchanged) — DERIVED+BUILD-VERIFIED (testing-welcome), no B200 board ran it; sm100 int8/blockwise + MoE/MXFP4/MLA legs are residual bricks**; Volta/Pascal need CUDA<13 | ACTIVE | **SECOND NON-GB10 RUNTIME PROOF: Orin sm_87 portable bf16 SYNC path RUNTIME-VERIFIED 2026-07-28 (`CLAIM-CUDA-ORIN-SM87-RUNTIME`)** — built portable-only for sm_87 on a real Jetson AGX Orin (CUDA-13 container BLOCKED by the 12.6 driver → l4t-jetpack r36.4.0 nvcc 12.6 + g++-13; all fast paths EMPTY for [87]), and `test_llama_paged_engine` (Llama-3.2-1B bf16) is **13/16 strict token-exact vs the vLLM 0.25.0 oracle, 16/16 near-tie gate, 0 divergent** on the sm_87 GPU (exceeds Thor's 12/16), + test_cuda_backend/test_cuda_ops 461 assertions of real on-device kernels; ONE honest sm_87 bug found — the DEFAULT async runner crashes (`cudaFree illegal memory access`), SYNC path (`VT_ASYNC_RUNNER=0`) is the verified one (unblock item). Residual Orin W-plan: fix async runner -> GGUF leg -> FA2 (needs cutlass) -> Marlin-int4 -> vs llama.cpp; Thor fp8/fp4 fast paths once SSH-reachable | | 10b | `ROAD-V1-D1-GDN-AOT` | **Per-arch Triton-AOT GDN packed-decode cubins (user-directed 2026-07-28):** the vendored Triton-AOT GDN fast-path cubins — the MEASURED codegen-win decode (Triton REG:205/0-spill vs hand-CUDA REG:255+STACK:48 spills) plus the delta_h/chunk_o/kkt/tril/wu FLA set — existed for **`sm_121a` ONLY**, so GDN-hybrid models (Qwen3.6) ran GDN decode correct but on the SPILLING hand kernel on every other arch (all cross-family arch builds ship `-DVLLM_CPP_TRITON=OFF`). Regenerate the full GDN AOT set per-arch via the sanctioned `-DVLLM_CPP_TRITON_REGEN=ON` pipeline and vendor each `sm_XX/` tree so a `-DVLLM_CPP_TRITON=ON` build selects the non-spilling FLA path additively (build-time cubin selection = `sm_${VLLM_CPP_CUDA_ARCHITECTURES}`, no code change). | [kernel matrix](kernel-matrix.md), [backend matrix](backend-matrix.md) | [per-arch Triton AOT](specs/triton-aot-per-arch.md) | `DONE (DERIVED+BUILD-VERIFIED)` | **LANDED 2026-07-28 (`CLAIM-TRITON-AOT-PER-ARCH`, base `308c312a`):** full GDN AOT set regenerated + vendored for `sm_80/86/89/90a/100a` on dgx GB10 (Triton 3.6.0 / ptxas 12.8, cross-compiled — no target board needed; 57 artifacts + MANIFEST per arch matching the `sm_121a` fileset), `cuobjdump` proves real per-target SASS (`sm=80/86/89/90/100`, decode REG 209–217/0-spill, under the hand-CUDA spill floor); builder-path configure selects + integrity-verifies each tree; `check-triton-aot-drift.sh` rc=0 across all six trees; `sm_121a` byte-untouched (SACRED 27B/35B gate structurally unchanged). **HONEST: DERIVED+BUILD-VERIFIED — no non-`sm_121` board runs a GDN model here, so decode parity on these arches is build-verified, NOT runtime-measured.** Residual: runtime GDN-model verification on Ampere/Ada/Hopper/DC-Blackwell silicon when reachable | | 11 | `ROAD-V1-D2` | Tensor/multi-GPU and MoE sequence parallelism | [engine matrix](engine-matrix.md), [coverage view §3](feature-matrix.md#3-parallelism--scale-out) | TP spec written; v0.25.0 non-DP MoE sequence-parallel path is inventoried as `PAR-SEQUENCE-MOE` but unspiked | `READY` | acquire 2-GPU target and claim Phase 0 mock/ABI; its execution trace determines whether sequence parallel is part of the first performance slice | diff --git a/.agents/specs/cpu-a76-q8-dot.md b/.agents/specs/cpu-a76-q8-dot.md index 477c0f81e..72379d7d9 100644 --- a/.agents/specs/cpu-a76-q8-dot.md +++ b/.agents/specs/cpu-a76-q8-dot.md @@ -141,7 +141,7 @@ gate; this row only closes the proven Q8 assembly/compiler gap. | W3 | **GREEN** scheduled AAPCS64 kernel | ABI/disassembly checks and direct tests green | | W4 | **GREEN with T4 residual** interleaved Pi PMU A/B | 3.66-5.08% win on M1/T1 and M128; M1/T4 −2.43% | | W5 | **GREEN** recursive Qwen A/B and checkpoint | exact output; TTFT improves; E2E tie; docs current | -| W6 | Whole-system/thread/competitor exhaustion | M1/T4 partition, BF16 GEMM, memory/concurrency and llama.cpp floor | +| W6 | **Competitor measured, speed OPEN**; whole-system/thread exhaustion | vllm.cpp 0.461x prefill / 0.653x decode+E2E vs llama.cpp, 24.2% less RSS; profile BF16 GEMM and M1/T4 partition, then concurrency | ## R4-R5 binding result @@ -166,9 +166,16 @@ only on Cortex-A76+DotProd; every other CPU retains the portable path. The immutable commands, hashes, disassembly and raw-file index are in [the R5 evidence](../../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). -W6 remains open for the M=1/T4 scheduler interaction, BF16-GEMM optimization, -peak memory, concurrency and the same-file llama.cpp floor. Those gaps keep -this row `GATING`; they do not undo the demonstrated assembly/compiler win. +W6's same-file llama.cpp measurement is now complete and the speed floor is +NOT MET. On the exact 17-input/64-output shape, vllm.cpp is 0.461x llama.cpp +prefill and 0.653x decode/E2E, while peak RSS is better at 2.841 vs 3.747 GiB. +A same-text 64-token greedy check is byte-identical. This turns the former +unknown competitor gate into a measured 2.17x prefill / 1.53x decode target; +BF16 GEMM profiling, the M=1/T4 scheduler interaction and concurrency remain +open. The [competitor evidence](../../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md) +records the reconstructed llama.cpp b9892 pin, commands and raw hashes. These +whole-model gaps keep the row `GATING`; they do not undo the demonstrated +assembly/compiler win. ## Risks and decisions diff --git a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md index c12637653..bea49ead3 100644 --- a/.agents/specs/rpi5-cortex-a76-cpu-optimization.md +++ b/.agents/specs/rpi5-cortex-a76-cpu-optimization.md @@ -223,7 +223,7 @@ matching llama.cpp quant/repack cases for any borrowed layout or kernel: | R3 | **GREEN** Qwen trace + recursive scope profiling | R1-R2 | reached-loop inventory and binding baseline below | | R4 | **GREEN** A76 C++/NEON/SDOT provider | R3 ranked evidence | exact operation/model output; 2.4x scalar kernel speedup | | R5 | **GREEN with named T4 residual** A76 assembly candidate | R4 plateau + proven compiler gap | 3.66-5.08% binding kernel win; recursive model non-regressing | -| R6 | Whole-system/thread/serving exhaustion | accepted R4/R5 stack | all lever dispositions, <1% residual model, llama.cpp floor | +| R6 | **llama.cpp measured, speed OPEN**; whole-system/thread/serving exhaustion | accepted R4/R5 stack | close measured 2.17x prefill / 1.53x decode gap, M1/T4 and concurrency; retain RSS win | R1-R3 are the first implementation checkpoint. R4/R5 split into separate kernel-row PRs if the changed code exceeds the one-row helper size cap; this PR @@ -268,15 +268,35 @@ The recursive model gate is non-regressing: assembly versus compiler SDOT has 1.55% lower median TTFT, neutral TPOT and 0.13% lower E2E. Against the portable arm it lowers E2E 2.67%. `auto` selects the assembly only on Cortex-A76+DotProd. M=1/T4 remains a measured negative at −2.43%, so R6 owns -that thread-partition interaction plus BF16 GEMM, peak memory, concurrency and -the same-file Pi llama.cpp floor. Full provenance and raw hashes are in the +that thread-partition interaction plus BF16 GEMM and concurrency. Full +provenance and raw hashes are in the [R5 evidence](../../docs/bench-evidence/rpi5-a76-q8-dot-20260806.md). +## R6 same-file llama.cpp checkpoint + +The Pi competitor floor is no longer unknown. Against a locally QEMU-built +official llama.cpp b9892 (`ee445f93d`), on the same Q8_K_XL bytes, four cores +and 17-input/64-output shape, vllm.cpp measures 12.81 tok/s prefill, 2.55 tok/s +decode and 2.46 tok/s output-equivalent E2E. llama.cpp measures 27.77, 3.91 and +3.77 tok/s: vllm.cpp ratios 0.461x / 0.653x / 0.653x. llama.cpp is therefore +2.17x faster in prefill and 1.53x faster in decode/E2E, well outside each +arm's sub-1% spread. vllm.cpp wins peak RSS at 2.841 vs 3.747 GiB, 24.2% less. +Both engines produce byte-identical normalized text for a same-prompt 64-token +greedy check. + +R6 remains open on speed and concurrency, not memory or correctness. The prior +profile already puts BF16 `Bt16Neon` at 57.76% of the model run, so the next +recursive step is a fresh clean profile of both engines followed by the BF16 +GEMM compiler/output audit; the M1/T4 Q8 partition remains a secondary leaf. +The exact b9892 reconstruction caveat, commands, samples and raw hashes are in +the [Pi competitor evidence](../../docs/bench-evidence/rpi5-a76-llamacpp-20260806.md). + ## Risks and decisions - `BACKEND-CPU` already passes a 20-core Arm i8mm Qwen3.5-2B single-stream llama.cpp floor. That evidence does not transfer to a four-core A76 without - i8mm; Pi numbers begin `PENDING` and never replace the existing scoreboard. + i8mm; the Pi now has its own measured open speed gate and never replaces the + existing 20-core scoreboard. - The Q8_K_XL file contains substantial f16 weights, so quant GEMM alone cannot be assumed dominant. A fresh trace ranks work before each lever. - Reordered floating reductions may move near ties. Exact integer paths and the diff --git a/.agents/state.md b/.agents/state.md index edffdbabc..7f5106eb8 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39438,3 +39438,31 @@ disassembly excerpts are indexed at M=1/T4 scheduler interaction, the still-dominant BF16 GEMM, peak memory, concurrency and the same-file Pi llama.cpp floor. No Pi competitor-parity claim is made. + +## 2026-08-06 — RPi5 same-file llama.cpp floor measured + +The four-core Cortex-A76 lane now has its independent same-file competitor +result. Official llama.cpp tag b9892 (`ee445f93d`) was built locally for +AArch64 under QEMU with GCC 13.3, DotProd+FP16, OpenMP and no accelerator +backend, then copied to the execution-only Pi. The historical project object +`237ad9b96` is unavailable from both recorded remotes; b9892's Q8/Arm/repack +and Qwen3.5 line anchors match the project record exactly, and the binding +evidence records this reconstruction plus the actual binary SHA. + +The vllm.cpp nominal input length 16 tokenized to 17 tokens. Three clean +vllm.cpp reps and llama.cpp pp17/tg64/pp17+tg64 (three timed samples after +warmups) were all unthrottled. Medians/means: vllm.cpp prefill 12.81 tok/s, +decode 2.55 tok/s, output-equivalent E2E 2.46 tok/s, E2E 26,018.39 ms; +llama.cpp 27.77 / 3.91 / 3.77 tok/s, E2E 16,998.49 ms. Ratios are 0.461x +prefill and 0.653x decode/E2E, so the Pi speed floor is NOT MET. vllm.cpp wins +peak RSS, 2.841 vs 3.747 GiB (24.2% less). A separate same-text 64-token greedy +CLI check is byte-identical after trailing-newline normalization, SHA-256 +`a5a630d7e9774c2300f5dda67a085d43ab1cf9125480c37208ae1c24a2eb25e0`. + +An initial `/proc` sampler that forked two `awk` processes every 50 ms is +explicitly VOID because it inflated load and slowed both arms. Accepted +throughput ran with no sampler; RSS ran separately with a once-per-second +shell-builtin `VmHWM` reader. Full commands, samples, pins and raw hashes are +in `docs/bench-evidence/rpi5-a76-llamacpp-20260806.md`. R6 now owns a measured +2.17x prefill / 1.53x decode speed gap, with fresh cross-engine profiling and +the already-dominant BF16 GEMM first; M1/T4 and concurrency remain open. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 090a0f557..48b905a76 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -8,7 +8,7 @@ | **vLLM** | Qwen3.6-35B-A3B NVFP4, GB10 | 0.93x to 1.03x: ahead at c4, worst c16 0.93x | identical | | **vLLM** | DeepSeek-V2-Lite (MLA), GB10 | 0.86x to 0.95x throughput, TTFT wins at c4/c8 | identical | | **vLLM** | Laguna-XS-2.1 NVFP4, GB10 | **parity+, 1.03x** (44.46 vs 43.10 tok/s, byte-exact, default config; bf16 weights now device-resident) | near-tie | -| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill **1.18x ahead**, decode tie, memory parity. RPi5/A76: assembly component gate **1.04-1.05x vs compiler SDOT**; same-file llama.cpp floor pending | byte-identical on both Arm lanes | +| **llama.cpp** | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill **1.18x ahead**, decode tie, memory parity. RPi5/A76: vllm.cpp is **0.461x prefill / 0.653x decode+E2E**, but uses **24.2% less RSS** | byte-identical on both Arm lanes | | **MLX-LM** | Qwen3-0.6B, Apple M4 | 97.6% warm total, prefill ahead | near-tie | | **DwarfStar** | DeepSeek-V4-Flash GGUF, GB10 | **beats ds4, 1.144x** (18.69 vs 16.33 tok/s, byte-exact, default config) | n/a, GGUF peer | @@ -175,8 +175,18 @@ lowers TTFT 33.40% and E2E 2.67%. Cortex-A76+DotProd therefore selects the assembly arm by default; every other CPU retains portable dispatch. Exact commands, binary/raw-file hashes, all four kernel arms and the disassembly are in the [immutable evidence index](bench-evidence/rpi5-a76-q8-dot-20260806.md). -Peak memory, concurrent serving and the same-file Pi llama.cpp comparison stay -`PENDING`; no Pi competitor-parity claim is made. +The same-file Pi competitor floor is now measured and **NOT MET on speed**. +The nominal 16-token vllm.cpp prompt measures 17 tokens, so the accepted +llama.cpp denominator is pp17/tg64/pp17+tg64. vllm.cpp reaches **12.81 tok/s +prefill, 2.55 tok/s decode and 2.46 tok/s output-equivalent E2E**, against +llama.cpp's **27.77 / 3.91 / 3.77 tok/s**: ratios **0.461x / 0.653x / +0.653x**. Median E2E is 26,018.39 vs 16,998.49 ms, so llama.cpp is 1.53x +faster overall. vllm.cpp wins peak RSS, **2.841 vs 3.747 GiB (24.2% less)**. +Three clean repetitions are stable, every leg is unthrottled, and a same-text +64-token greedy CLI check is byte-identical after trailing-newline +normalization. The [binding evidence](bench-evidence/rpi5-a76-llamacpp-20260806.md) +records commands, pin reconstruction, raw hashes, and the discarded intrusive +sampler run. Pi concurrency remains `PENDING`. Same GGUF file both arms, `dgx.casa` GB10 aarch64 (20 cores), idle, 3 reps, llama.cpp `237ad9b96` built fresh on the same host. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 096dc24ae..d1bc9021e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -152,7 +152,7 @@ the registered engine forward. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; A76 Q8 SDOT assembly gated and default) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | ☐ | ✅ | ✅ | ✅ | diff --git a/docs/STATUS.md b/docs/STATUS.md index d668d0c5b..ba12eb41a 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -86,7 +86,7 @@ token-for-token correctness against the pinned oracle. | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | -| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 Q8 assembly `GATING` | Binding 20-core arm: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi: QEMU-built AAPCS64 leaf is output-exact and beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4; 64-token Qwen TTFT −1.55% and E2E −0.13% vs SDOT. M1/T4 is −2.43%, so thread partition, peak memory, concurrency and same-file Pi llama.cpp floor remain open | +| CPU backend vs llama.cpp | 20-core Arm/i8mm at floor; RPi5/A76 speed below floor, `GATING` | Binding 20-core arm: prefill 1.18x, decode/RSS parity, tokens byte-identical. Pi: the output-exact AAPCS64 leaf beats compiler SDOT 3.66-5.08% on M1/T1 and M128/T1/T4, but same-file llama.cpp is 2.17x faster in prefill and 1.53x faster in decode/E2E (vllm.cpp ratios 0.461x/0.653x); vllm.cpp uses 24.2% less peak RSS and same-text 64-token greedy output is byte-identical. M1/T4 remains −2.43%; BF16 GEMM, thread partition, speed closure and concurrency remain open | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | diff --git a/docs/bench-evidence/rpi5-a76-llamacpp-20260806.md b/docs/bench-evidence/rpi5-a76-llamacpp-20260806.md new file mode 100644 index 000000000..5294c3782 --- /dev/null +++ b/docs/bench-evidence/rpi5-a76-llamacpp-20260806.md @@ -0,0 +1,117 @@ +# Raspberry Pi 5 CPU vs llama.cpp, 2026-08-06 + +Binding same-file comparison for the Raspberry Pi 5 Cortex-A76 arm of +`BACKEND-GATE-CPU-LLAMACPP`. Both engines were built locally for AArch64 under +QEMU and copied to the Pi for execution. Nothing was compiled on the Pi. + +## Result + +The implementations produce the same greedy text, but vllm.cpp does not yet +meet the llama.cpp speed floor on this four-core target. llama.cpp is 2.17x +faster in steady-state prefill and 1.53x faster in decode and combined +prompt-plus-generation time. vllm.cpp uses 24.2% less peak resident memory. + +One vllm.cpp request, `--input-len 16 --output-len 64`, tokenized to 17 input +tokens. The binding llama.cpp arm therefore uses `pp17`, `tg64`, and +`pp17+tg64`, not the nominal 16-token target. vllm.cpp medians are three clean +process repetitions. llama.cpp reports three in-process timed samples after +its standard warmups. Every accepted leg was unthrottled. + +| Axis | vllm.cpp | llama.cpp | vllm.cpp / llama.cpp | +|---|---:|---:|---:| +| Prefill | 12.81 tok/s (`17 / TTFT`) | **27.77 tok/s** | **0.461x** | +| Decode | 2.55 tok/s (`1000 / TPOT`) | **3.91 tok/s** | **0.653x** | +| Output-equivalent E2E | 2.46 tok/s (`64 / E2E`) | **3.77 tok/s** | **0.653x** | +| E2E latency | 26,018.39 ms | **16,998.49 ms** | **1.531x latency** | +| Peak RSS | **2.841 GiB** | 3.747 GiB | **0.758x, 24.2% less** | + +The vllm.cpp median TTFT is 1,327.25 ms and median TPOT is 391.93 ms. +Its three E2E samples are 25,946.07, 26,018.39 and 26,021.14 ms, a 0.29% +range. llama.cpp's p17 result is 27.7681 +/- 0.0320 tok/s, tg64 is +3.9091 +/- 0.0142 tok/s, and combined p17+tg64 takes +16,998.49 +/- 25.13 ms. This is a measured speed gap, not run noise. + +## Correctness + +The three vllm.cpp performance repetitions emitted the existing 64-token +golden byte-for-byte, SHA-256 +`0ec98eabb23e4148d540fcf79a2fe61678fb90fe462cdf28134af7a42fe6a826`. + +An additional raw-prompt check ran both CLIs with the same text, greedy +sampling and 64 output tokens. Both tokenizers measured 24 prompt tokens and +both completed 64 tokens. The files differ only in trailing line endings; +after removing trailing whitespace both have SHA-256 +`a5a630d7e9774c2300f5dda67a085d43ab1cf9125480c37208ae1c24a2eb25e0`. +Thus the competitor gap is performance-only for the checked stream. + +## Provenance + +- Pi: Raspberry Pi 5, four Cortex-A76 r4p1 cores, DotProd present, i8mm + absent, 8 GiB RAM, Debian kernel `6.18.34+rpt-rpi-2712`. +- Model: `Qwen3.5-2B-UD-Q8_K_XL.gguf`, 2,823,978,240 bytes, SHA-256 + `a53988df91157d78acaf3c95e22db179d13f6236061bdb86576494dc99b1bc3b`. +- vllm.cpp: branch head `9044c2a7d`; assembly-default `vllm-bench` SHA-256 + `9eb57cf3760eaade9dcef03dda1648556577c44199369ad38bf42083efbc70a9`. +- llama.cpp: official tag `b9892`, commit + `ee445f93d8a0a5033a46d1960e901ef5caec9a41`; `llama-bench` SHA-256 + `d9d93d8b38d0d8faa676f7d48f1a8fcbbc235f3a5697fc2ac787422d61783d52`. + Ubuntu 24.04/GCC 13.3, `GGML_NATIVE=OFF`, + `GGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16`, OpenMP on, accelerator + backends off. + +The historical project record calls llama.cpp pin +`237ad9b961f009ae19ac29dbce4cd0c1251f94b3` “b9892”. That object is no longer +available from the recorded fork or official remote, so it could not be +reproduced byte-for-byte. The official b9892 tag above is the binding +reconstruction. Its recorded anchors match exactly: portable Q8 dot +`ggml/src/ggml-cpu/quants.c:400`, Arm Q8 dot +`ggml/src/ggml-cpu/arch/arm/quants.c:1076`, Q8 repack +`ggml/src/ggml-cpu/repack.cpp:2725`, and `src/models/qwen35.cpp`. The exact +commit and binary hash are recorded so this substitution is explicit rather +than silently attributed to the unavailable object. + +## Commands + +vllm.cpp, repeated three times in the interleaved clean series: + +```sh +taskset -c 0-3 env VLLM_CPP_CPU_THREADS=4 VT_CPU_Q8_DOT=auto \ + bin/final/vllm-bench \ + --model models/Qwen3.5-2B-UD-Q8_K_XL.gguf \ + --num-prompts 1 --input-len 16 --output-len 64 --concurrency 1 \ + --seed 0 --temperature 0 --output-token-ids evidence/out.json +``` + +llama.cpp, three timed repetitions in one loaded process: + +```sh +taskset -c 0-3 bin/llama-b9892/llama-bench \ + -m models/Qwen3.5-2B-UD-Q8_K_XL.gguf \ + -p 17 -n 64 -pg 17,64 -t 4 -r 3 -o jsonl --progress +``` + +Peak RSS was measured in separate same-workload passes from Linux `VmHWM`, +sampled once per second with shell built-ins. The sampler does not run in the +timed performance series. An earlier attempt that forked two `awk` processes +every 50 ms is `VOID`: it inflated load and slowed both engines. A clean p16 +cross-process series was also non-binding once vllm.cpp reported that the +nominal prompt actually contained 17 tokens; it was retained only as a +corroborating diagnostic. + +## Raw evidence hashes + +| Evidence | SHA-256 | +|---|---| +| Clean environment log | `ea2955cd99843645ff4c36d293e0fd61e49910b184e81e04f7f29888c4c3cecd` | +| Clean manifest | `3cf4c33c86ad938ff7df1f84f9db38a5ca198a6fb9cf4513e2640d00dc16320c` | +| vllm.cpp outputs, reps 1/2/3 | `07fcd93a0953d258af36e5c5ccd30ee215e399e50ac3d81e6efd8e551812479d` / `3ef8ca5e5497f2f68aeb8a4f74db3db7bfdf18bab4dc333e7437d17a9c03c060` / `1c796eac3e947dc474c7311bae7d5a64a223fe767c9569b4750ae5a99b7f60ce` | +| vllm.cpp RSS record | `426f2ef1c000da16a228f099dd3f1daf896c14a6f6ae1dcc4f6a5843e5457a27` | +| llama.cpp p17 JSONL | `7960035bd312117efa8f9c208c4dc53207d7379af42d9462c5a3c1add5b5e83f` | +| llama.cpp p17 environment | `13f07504d373558ffa74d57f5d2ca8f4453c935b4b3b78268f748ebc89a19fc6` | +| llama.cpp p17 RSS record | `5b00b4bfd1a13a6a0073452095278934cf9be57ea32fa1a70583a285c36ad1b3` | +| Exact-prompt vllm.cpp stdout | `8e0a7064f2b61462926ee554d3d15f3c5b1297c0ff1873cbc0e6391fde498097` | +| Exact-prompt llama.cpp stdout | `490e2379acd27dccf55dbdb0485052f4424a5af6654b6862e675adfaa172b67d` | +| Exact-prompt environment | `ed75ef54044549d379426472e0d935a3710a14049ec4321662492a207fc9bfc3` | + +Raw files remain under +`rich@rpi5fan.lan:~/vllm-cpp-assembly/evidence/llama-compare-20260806/`. diff --git a/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md b/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md index 9ebce2f6b..cec2cc3a3 100644 --- a/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md +++ b/docs/bench-evidence/rpi5-a76-q8-dot-20260806.md @@ -16,8 +16,9 @@ One residual is retained rather than hidden: M=1 at four threads is 2.43% slower than compiler SDOT despite retiring fewer instructions. That small fixture is dominated by the enclosing threadpool schedule; the recursive 64-token model gate is neutral on TPOT and non-regressing end to end. The row -therefore remains `GATING`, with the M=1/T4 partition and the same-file -llama.cpp floor still open. +therefore remains `GATING`, with the M=1/T4 partition and whole-model speed +closure still open. The subsequently measured same-file llama.cpp floor is +indexed separately below. ## Provenance @@ -141,5 +142,8 @@ VLLM_CPP_CPU_THREADS=4 VT_CPU_Q8_DOT=a76-asm bin/vllm-bench \ Replace `a76-asm` with `sdot` and `portable`; verify `sha256sum` and `vcgencmd get_throttled` before and after every arm. QEMU timings are not -performance evidence. The same-file llama.cpp comparison remains pending and -no competitor parity claim is made here. +performance evidence. The same-file llama.cpp comparison was subsequently +completed and is **NOT MET on speed**: see +[the Pi competitor evidence](rpi5-a76-llamacpp-20260806.md). This document +remains the assembly/compiler A/B record; the competitor record owns its +independent binaries, commands and raw hashes.