feat(SPEC-DFLASH2): the grouped dynamic depthwise convolution, reached by a DFlash2 draft that now refuses at the selector (#1314)

.agents/claims/CLAIM-SPEC-DFLASH2-W2.md

@@ -0,0 +1,5 @@
+# CLAIM-SPEC-DFLASH2-W2
+
+| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update |
+|---|---|---|---|---|---|---|---|
+| `CLAIM-SPEC-DFLASH2-W2` | `SPEC-DFLASH2` (`ACTIVE`), `KERNEL-DFLASH2-GROUPED-CONV` (`ACTIVE`) | Claude Code (opus-5), helper role — fresh implementer working from the committed spec `.agents/specs/dflash2-spec-decode.md` | isolated worktree, CPU only; no GPU, no `nvcc`, no oracle run, no lease, no checkpoint download | `row/SPEC-DFLASH2-W2`, issue [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | Owns ONLY wave W2 of the spec's `## Work breakdown`, plus the spec corrections issue [#1327](https://github.com/mudler/vllm.cpp/issues/1327) names: the `vt::DFlashGroupedConv` op (`OpId::kDFlashGroupedConv`, `DFlashGroupedConvArgs`, the CPU reference and the CUDA mirror) and its kernel-matrix row; the conv weights on `Qwen3DFlashConvWeights` / `Qwen3DFlashLayerWeights` / `Qwen3DFlashWeights` and their load in `LoadQwen3DFlash`; `DflashConvPrepare` / `DflashConvFinish` / `CheckDflashConvBatch` and their call sites in all three `Qwen3DFlashModel` layer bodies; the spec's `## Owed` O3 and O4 repairs in `MakeQwen3DFlashDraftConfig` (the `rope_parameters` and `dflash_config.block_size` fallbacks, the optional `layer_types`, and the `attention_sink_bias` refusal); the narrowing of `RefuseDflash2Draft` into `CheckDflash2DraftArm` and the new `RefuseDflash2CandidateSelector` with its two call sites; the conv's block from the resolved `k` in `LoadDflashDraft`; the three suites `tests/vt/test_ops_dflash2_grouped_conv.cpp`, `tests/vllm/models/test_qwen3_dflash2_draft.cpp` and `tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp`, and the updates to `tests/vllm/entrypoints/test_dflash2_draft_routing.cpp` the narrowing forces; the `SPEC-DFLASH2` engine-matrix row; the `KERNEL` count in `scripts/check-agent-record.py`; and the spec's `## Scope`, `## Upstream chain`, `## Gates`, `## Risks/decisions` D9/D10, `## Owed` and `## Now`. EXCLUDES the candidate selector and its top-k (W3), the speculator and its device path walk (W4), the GGUF drafter ARM (W5), and the G1-G5 run gates (W6). EXCLUDES any parity-pin advance and any DFlash1 behaviour beyond the config-builder fallbacks, which are additive and are asserted to leave every published DFlash1 draft byte-for-byte unchanged | `ACTIVE` | 2026-08-19 — W2 landed red-first; every added behaviour mutation-proven with its compile status and a sha256-verified byte-for-byte restore; two gate weaknesses found BY the mutation pass and repaired before landing; CUDA arm written but never compiled (no `nvcc` on this host) and recorded as spec `## Owed` O6 |

.agents/issue-index.md

@@ -469,6 +469,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1394](https://github.com/mudler/vllm.cpp/issues/1394) | `ENG-CUDAGRAPH-BREAK` | The CPU paged attention reads `btab[r * bt_row + (j / block_size) * bt_col]` for every `j < seq_lens[r]` without checking that the block table has that many columns, so a caller with a short table gets an out-of-bounds read, a plausible block index out of it, and attention over the WRONG page — silently. `tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp`'s `SpecAttnMeta` supplies one: hardcoded `block_table_num_cols = 1` against shape C's `seq_lens = 24` at `block_size = 16`. Found while fixing [#1380](https://github.com/mudler/vllm.cpp/issues/1380), whose `DevicePool` change moved the bytes after the table and turned the same read into a SIGSEGV on `thor:gpu0` (`gdb` at `src/vt/cpu/cpu_paged_attn.cpp:224` under `FullAttnBlockPaged`). PRE-EXISTING: the case passes at `origin/main` only because the read landed on bytes that decoded to an in-range index. FIXED IN FLOW in both halves — the kernel refuses a short table with one compare per request outside the token loop, and the helper sizes its table for the sequence length it declares. Owned by row `ENG-CUDAGRAPH-BREAK` | bug |
 | [#1387](https://github.com/mudler/vllm.cpp/issues/1387) | `ENG-EXPERT-STREAM-DEVICE` | `docs/FEATURES.md`'s routed-expert-streaming row still read "CPU keep-quant towers only" after W0c made the seam take the slot arm on `is_cpu()` OR `host_memory_is_device_addressable()` and W0d made the load-time fit refusal drop those towers from its bound. `AGENTS.md` routes a feature-surface change to that page, and the change that moved the surface did not write it. `scripts/check-doc-checkpoint.py` said so, on commit `939755f99` of `row/ENG-EXPERT-STREAM-DEVICE-W0`: a measurement was appended to `.agents/benchmark-record.md` with no `docs/FEATURES.md` edit beside it. The PAGE is FIXED IN FLOW while repairing the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377): the row now names both the device arm ([#1124](https://github.com/mudler/vllm.cpp/issues/1124)) and the residency condition ([#1378](https://github.com/mudler/vllm.cpp/issues/1378)), and the observability detail it displaced to stay inside the 220-character cell budget is stated at `docs/USAGE.md:4598-4620`. The GATE is NOT fixed and needs a decision: the checker walks a range one COMMIT at a time, so once a commit is published on a branch that may not be force-pushed, no later commit can make it green, and `scripts/agent-preflight.sh` keeps reporting `doc-checkpoint range` red on this branch until it merges. The squashed commit that lands on `main` carries both paths and passes. Changing the walk is checker semantics and needs its own row, spec and red-first evidence per `AGENTS.md` "Changing the rules or a checker", so it is not folded in here. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug |
 | [#1386](https://github.com/mudler/vllm.cpp/issues/1386) | — | `tools/bench/gpu_clock_state.py`'s `QUERY_FIELDS` collects nine fields and **none of them is thermal and none is electrical**, so the driver's own `SwThermalSlowdown` label can never be checked against a die reading on any window this helper has ever recorded. The measured consequence is that the nine windows of 2026-08-19 cannot distinguish a load transition from a thermal excursion. The concrete evidence is `clock-c1-r1.samples.json` in `/mnt/nas_share/rc/q38bf16/out/bench-20260819T035148Z/`: ours c1 r1 dips five times on the same period at the same `utilization.gpu = 96` — 48.83 s / 2177 MHz, 80.60 s / 2320 MHz, 109.28 s / 2210 MHz, 137.98 s / 2359 MHz, 166.07 s / 2268 MHz — and **two of those five carry `0x0000000000000000`**, no throttle bit at all (2210 and 2359), while three carry `0x20`. The 2210 MHz unlabelled dip is deeper than two of the three labelled ones, so the driver labels comparable excursions inconsistently and the bit alone cannot decide it. What would settle it: add `temperature.gpu` and `power.draw` to `QUERY_FIELDS`. That changes the clock-record schema, so it owes its own row and spec. Split out of [#1354](https://github.com/mudler/vllm.cpp/issues/1354) and owed under `## Owed` in [lease-clock-pinning.md](specs/lease-clock-pinning.md) | gap |
+| [#1327](https://github.com/mudler/vllm.cpp/issues/1327) | `SPEC-DFLASH2` | `.agents/specs/dflash2-spec-decode.md` `## Upstream chain` said the three output scalars `input_embedding_scale`, `output_multiplier` and `final_logit_softcapping` are "ABSENT from this config" and that "no published checkpoint exercises them, so the port implements them and gates them synthetically". That was measured on `z-lab/Qwen3.8-27B-DFlash2` alone. `z-lab/Muse-Glimmer-30B-DFlash2` — the SECOND published DFlash2 checkpoint, `config.json` sha256 `cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9`, read 2026-08-19 — sets `output_multiplier` to `0.19611613513818404` and `final_logit_softcapping` to `20.0`, and ships `block_size` 16 against the 27B's 8, hidden 6656 (416 groups, a 1664-wide `kernel_projection`) and `rope_theta` 500000.0. Both scalars are applied to the candidate VALUES in `compute_candidates` BEFORE the selector scores them, so a wrong value reorders the top-K and moves acceptance without raising — the `is_causal` failure class one layer up, and the class no token gate here can see. A port reading all three with `.get(key, default)` would pass every gate built from the 27B draft and be measuring the default path. The same file also falsifies `## Scope`'s exclusion of "a second DFlash2 target family": upstream registers ONE architecture and both checkpoints declare `model_type` `qwen3`, so what the second adds is values rather than a class. FIXED IN FLOW by `SPEC-DFLASH2` W2, which is the wave that had to read both configs anyway: `## Scope` drops the exclusion, `## Upstream chain` records both values with their source, `## Gates` G1 now requires BOTH published block shapes (upstream's own reference test parametrises 5 and 8 and never reaches 16), and `## Risks/decisions` D9 binds W3 to gate the scalars against the checkpoint that sets them | gap |
 | [#1413](https://github.com/mudler/vllm.cpp/issues/1413) | `LTX25-DEVICE-RESIDENCY` | The LTX-2.5 render phase table [#1408](https://github.com/mudler/vllm.cpp/pull/1408) lands for [#1010](https://github.com/mudler/vllm.cpp/issues/1010) is written by the SUCCESS PATH ONLY — `WritePhaseLog` sits after `im.trace.completed = true` at `src/vllm/multimodal/ltx2_video.cpp:4655-4658 @ 4f3c24380` — so a render that is killed, aborted by a lease governor, or still running writes no table at all. That is the population the campaign actually has: [#1375](https://github.com/mudler/vllm.cpp/issues/1375) is `ABORT[92] PROJECTED OVERRUN`, `child exit=-15`, 0 frames; [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) rung 1 is `EXIT=137`, 0 frames; rung 2 is `EXIT=1`, 0 frames. **And nothing is emitted WHILE a render runs**: `PhaseLog::Open`/`Close` print nothing and `VLLM_RENDER_PHASE_LOG_STDERR` fires inside `WriteJson`, i.e. on the success path again, so between `ltx2-gen: family=...` and `wrote N frames` a 2.5-hour render is silent and working is byte-identical to hung. **The unit that costs the wall has no counter**: `denoise` is one leaf covering ~2.7 h, against #1375's measured ~162 s per DiT forward at 60 structural forwards (30 steps x 2 CFG legs — `cfg_scale != 1.0` forces the unconditional branch at `ltx2_pipeline.cpp:521-523`). External sampling is NOT the fallback and was tried: #1375 records `phase=OTHER` throughout because `eu-stack` unwinds ZERO frames inside the `rc` worker container, and its own text names an in-process phase marker as the way to attribute the 162 s. Owed: a stderr line on every phase open and close, so the LAST LINE PRINTED names the phase in flight, plus one tick per DiT forward carrying phase, step `k/N`, cumulative forward index, elapsed and seconds-since-previous — on the shipped default, because the failure happened on default settings and `VT_H3_PROGRESS` (`minimax_h3.cpp:776-793`) is the opt-in shape that is exactly why no LTX-2.5 run has one. Precondition for attributing #1375, which caps render resolution. Stage W0-live in [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) | feature |
 | [#1320](https://github.com/mudler/vllm.cpp/issues/1320) | `ENG-EXPERT-STREAM-DEVICE` | `VT_CPU_QUANT_REPACK` rewrites a Q8_0 weight into the `block_q8_0x4` i8mm interleave at load, and only the CPU `MatmulBTKernel` understands that layout — the CUDA quant dot reads plain `block_q8_0` and never consults `Tensor::repacked`, so a repacked weight uploaded to a device produces WRONG TOKENS rather than a crash. Its sibling `elem_kn_repack` has BOTH a CPU-platform gate in the loader policy (`gguf_keep_quant.cpp`, `CurrentPlatform().device_type() == kCPU`) and a refusal in `ResidentWeight`; `quant_repack` had NEITHER, because it rides `vt::cpu::QuantRepackActive()`, a HOST-CPU Arm i8mm probe that says nothing about where the weight will execute. An aarch64 box doing `--device cuda` — which is exactly `dgx:gpu0`, a Grace-Blackwell GB10 — satisfies it. Currently silent rather than firing, and that is measured: eligibility is Q8_0 with `n % 4 == 0 && k % 32 == 0`, `Qwen3.8-2.4T-A95B UD-Q1_0` has exactly one Q8_0 tensor at 0.01% of parameters, and the instrumented load of that checkpoint recorded `quant_repack = 0`. Found while implementing W0f ([#1299](https://github.com/mudler/vllm.cpp/issues/1299)), which changes the same function and whose G0-CORRECT gate compares CUDA tokens against CPU tokens — precisely the instrument that would have reported this as a W0f defect. FIXED IN FLOW as the tripwire its sibling already has, placed beside it so it covers the staging branch AND W0f's aliasing branch (where the bytes live was never the question; the kernel that reads them is), gated red-first on both arms and mutation-proven. `VT_CPU_QUANT_REPACK=0` is the operator's way past it. NOT fixed here, deliberately: adding the missing CPU-platform gate to `p.quant_repack` itself is a loader-policy semantics change owned by `QUANT-GGUF-KEEPQ-LOADER`, carried under `## Owed` in [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug |
 | [#1426](https://github.com/mudler/vllm.cpp/issues/1426) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 x0 model picks its DiT forward with one ternary on `im.on_device` (`src/vllm/multimodal/ltx2_video.cpp`), and **no test on a box without an accelerator enters the DEVICE branch**, so dropping its `perturbations` argument is undetectable. MEASURED, both mutations built clean at `compile_err=0` and both restored byte-for-byte with the target rebuilt: **D10** (drop `p` from the device branch alone, 1 insertion / 1 deletion) is **GREEN**, `test_ltx2_video` exit 0, 88 cases / 2755 assertions; **D11** (drop it from BOTH branches, 3 insertions / 2 deletions) is **RED**, exit 1, 2 cases / 3 assertions, failing `CHECK(MaxAbsDiffOf(t.video_first_perturbed, t.video_first_cond) > 1e-6 * latent_span)`, the same for `video_first_modality`, and `CHECK(n.video_first_perturbed != n.video_first_cond)`. D11 also has to be written with `(void)p;` to be a RESULT at all, because the naive form leaves the lambda parameter unused and `-Werror=unused-parameter` refuses the build - a mutation that fails to build reads exactly like a passing test. This is a missing RUNNER and not a missing test: `Ltx2VideoEngine::Load` refuses `device != 0` in three places (`ltx2_video.cpp:749-758`, `:779-788`, and the queue-creation check below them) unless a non-CPU platform resolves, registers a backend and does not decline `kLtx2VideoFamily`, and `.github/workflows/ci.yml` has no GPU runner. A SYNTHETIC accelerator is DEFERRED rather than impossible, and the PARTS ALREADY EXIST. `vt::RegisterOp(OpId, DeviceType, void*)` is public (`include/vt/op_provider.h:127`) and is exactly how `src/vt/cpu/cpu_ltx2.cpp:201-205` installs the CPU glue table, so a test can read that pointer back with `vt::GetOp(OpId::kLtx2, kCPU)` and register it for a second device type; `RegisterReferenceTier` loops EVERY OpId (`src/vt/op_provider.cpp:795-807`) and `MaybeInstallReferenceTier` copies `src->fn` verbatim (`:204-225`), which covers the `vt::` ops the staging and the forward need; `Glue()` `VT_CHECK`-refuses BY NAME and never returns nullptr (`src/vllm/model_executor/models/ltx2_device.cpp:81-86`); and the closed 7-value `DeviceType` enum (`include/vt/device.h:16-28`) is handled the way `tests/vllm/multimodal/test_diffusion_device_seam.cpp` ALREADY handles it FOR THIS FAMILY — a SEPARATE EXECUTABLE (`:19-23`, `tests/CMakeLists.txt:317`) that registers a unified-memory `FakeXpuBackend` (`:60-73`), a platform in the XPU and CUDA slots (`:116-121`) and a `device = 1` `kLtx2VideoFamily` load (`:127-133`) — which leaves `test_ltx2_video`'s 88-case baseline byte-for-byte untouched. One design fact goes with that: `Ltx2DeviceKernelsAvailable` reads `vt::OpRegistered`, which EXCLUDES the reference tier deliberately (`op_provider.cpp:677-701`, "a FALLBACK, not a native kernel"), so the glue table needs the direct `RegisterOp` and the tier serves the rest. A fake also DOES prove link B as this row defines it — the x0 model reaching `Ltx2DitForwardDevice(..., p)`, which would turn D10 RED; what it cannot prove is that a GPU runs it, which is the hardware half the leased run below closes. WHAT IT COSTS is that new executable plus a fixture DiT staged through the fake backend (`tests/vllm/multimodal/ltx2_video_fixture.h` already writes a complete reduced model). TWO gates on the route were checked and pass by construction: `vt::RegisterBackend(kXPU, ...)` writes slot 0 (`src/vt/backend.cpp:81-84`) so `:800`'s `TryGetBackend(Device{kXPU, 0})` resolves for `params.device = 1`, and `vt::CreateQueue(Device)` takes its `entry.backend->CreateQueue()` arm (`backend.cpp:146-153`) whose `q.device == device` assertion the fake satisfies. TWO are NOT verified and NOBODY HAS BUILT IT: whether `Ltx2StreamDitToDevice` (`:847`) stages the checkpoint onto that queue, and whether the bf16 render that follows (`:812`) completes through the reference tier. PLAUSIBLE, NOT PROVEN — recorded in those terms because the first draft of this row asserted the opposite three facts as measured. What IS gated: `Ltx2DitForwardDevice` honours all four perturbation fields on the CPU backend, nine mutations RED (`tests/vllm/models/test_ltx2_device.cpp`). Closes on one leased `one_stage --device cuda` run at the model's own guider defaults recording `dit_forwards == 4 * dit_evaluations`. Same shape as [#1131](https://github.com/mudler/vllm.cpp/issues/1131). Listed under `## Owed` in [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) §12.8 link B | bug |