From 733ad8316fe4fdbd4315b5617f29052d393276e0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 22:26:56 +0000 Subject: [PATCH 1/3] feat(SPEC-DFLASH2): land the grouped dynamic convolution and make a DFlash2 draft run it (#1314, #1327) Wave W2 of `SPEC-DFLASH2` ([#1314](https://github.com/mudler/vllm.cpp/issues/1314)). W1 shipped a refusal; this wave ships the first of the two mechanisms it named, and moves the refusal so that mechanism is REACHED. **`vt::DFlashGroupedConv` is the project's first dynamic, grouped, block-masked convolution.** `out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, with tap `t` live only where `(i mod (1+k)) >= t`, `g(c) = c / conv_group_size`, and `base_kernel` dim 0 the prepare/finish SIDE rather than a tap. Three things separate it from the shipped `KERNEL-DEPTHWISE-CONV1D`: the weights are projected per position from the sublayer input rather than static, one delta serves a GROUP of channels while the base is per channel, and the tap mask is over the query BLOCK rather than causal over the sequence -- which is what lets a proposal position see the ones before it without another backbone pass. The CPU kernel is the authoritative reference and rounds to the tensor dtype after each step, as upstream's bf16 chain materializes it, so the op is elementwise with no reduction-order freedom and the CUDA mirror is specified BIT-IDENTICAL rather than within an envelope. Both of upstream's position-mask arms are ported (`pos & (block-1)` and `pos % block`) and gated at block 5, 8 and 16. The op lands as kernel-matrix row `KERNEL-DFLASH2-GROUPED-CONV` (`ACTIVE`, `CLAIM-SPEC-DFLASH2-W2`), which is why the `KERNEL` count in `scripts/check-agent-record.py` moves 52 -> 53. It stays `ACTIVE` rather than `DONE` because its CUDA arm has never compiled here. **The refusal moved so the conv could be reached.** A safetensors `DFlash2DraftModel` draft is now ADMITTED at `CheckDflash2DraftArm`, loads its `attention_conv`/`mlp_conv` tensors, runs the conv in all three of the draft's layer bodies -- including the paged body the production decode path reaches through `ForwardBlockLogitsWithDeviceKV` -- and is refused BY NAME at `RefuseDflash2CandidateSelector`, after the block forward and before anything samples. Keeping the startup refusal would have left every line of this wave unreachable from any production entry point, which AGENTS.md `## Nothing lands dead` forbids. A startup NOTICE names the boundary so the later refusal is not a surprise. The GGUF arm keeps its startup refusal, because its drafter arm has no conv weight path at all and admitting the file would load a DFlash1 draft out of a DFlash2 checkpoint. Recorded as spec `## Risks/decisions` D10. **Two `## Owed` blockers are discharged, and they were blockers.** `MakeQwen3DFlashDraftConfig` could not parse EITHER published DFlash2 `config.json`: it did `c.at("rope_theta")` and `c.at("block_size")` while both drafts nest them under `rope_parameters` and `dflash_config` (O3). It also required `layer_types`, which `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` does not declare, while upstream reads `getattr(config, "layer_types", None)` (O4). Both are FALLBACKS, tested after the flat spelling, so every published DFlash1 draft resolves byte-for-byte as before. O4 lands with a NAMED REFUSAL for `dflash_config.attention_sink_bias`: upstream passes a per-head sink into its `Attention` and this lane has none, so the parse repair alone would have turned a loud `key 'layer_types' not found` into a draft that loads with the sinks silently absent -- acceptance-only and invisible to a token gate. **#1327 is corrected here.** `## Upstream chain` claimed no published checkpoint exercises `input_embedding_scale`, `output_multiplier` or `final_logit_softcapping`. `z-lab/Muse-Glimmer-30B-DFlash2` sets `output_multiplier 0.19611613513818404` and `final_logit_softcapping 20.0`, and ships `block_size` 16 against the 27B's 8. Both are applied to candidate VALUES before the selector scores them, so a wrong one reorders the top-K and moves acceptance without raising. `## Scope`'s exclusion of a second DFlash2 target family is dropped (upstream registers ONE class; both checkpoints declare `model_type` `qwen3`), G1 now requires both block shapes, and D9 binds W3 to gate the scalars against the checkpoint that sets them. **Red before green, and two gate weaknesses found by the mutation pass.** The conv suite's red-before is a build failure (the op did not exist), exit 2. The draft suite's red-before is 5 cases / 4 failed, exit 1, with `[json.exception.out_of_range.403] key 'rope_theta' not found` -- O3 exactly as the spec predicted it. Eleven mutations then turn the focused suites red, each with its compile status printed and each restored byte-for-byte and verified by sha256. TWO of them came back GREEN first and are recorded rather than hidden: activating both convs at once could not tell one missing call site from none, and the first side probe could not see `args.side` forced to 0. Both gates were repaired before landing, and the second repair needed a fixture fix too -- an "inactive" taps=2 conv with an all-ones base is `x[i] + x[i-1]`, not the identity. **What is NOT reached, named as AGENTS.md `## Nothing lands dead` requires.** Three mutations stayed green and are `## Owed` in `.agents/specs/dflash2-spec-decode.md`, owned by row `SPEC-DFLASH2` under [#1314](https://github.com/mudler/vllm.cpp/issues/1314). O5: `LoadDflashDraft`'s own `conv_block_size = k + 1` is ungated, because that function is `static` in the loader's anonymous namespace and an entry-point gate on it must load a draft off a live target; owner W4. O6: the CUDA arm has NEVER COMPILED -- the authoring host has no `nvcc`, so the CUDA==CPU bit-identity case reports `no CUDA backend; skipping` and all 9410 assertions in that file are CPU; owed to a GPU lease. O7: `RefuseDflash2CandidateSelector`'s second call site, `GPUModelRunner::propose_drafts_block`, is not gated, and it is the one a user arrives through; owner W4. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/claims/CLAIM-SPEC-DFLASH2-W2.md | 5 + .agents/engine-matrix.md | 4 +- .agents/issue-index.md | 1 + .agents/kernel-matrix.md | 12 +- .agents/specs/dflash2-spec-decode.md | 408 ++++---- docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 2 +- .../vllm/model_executor/models/qwen3_dflash.h | 47 + .../gpu/spec_decode/dflash/speculator.h | 27 + include/vt/ops.h | 70 ++ scripts/check-agent-record.py | 15 +- src/vllm/entrypoints/model_loader.cpp | 97 +- .../model_executor/models/qwen3_dflash.cpp | 151 +++ .../models/qwen3_dflash_weights.cpp | 173 +++- src/vllm/v1/worker/gpu/runner.cpp | 8 + .../gpu/spec_decode/dflash/speculator.cpp | 25 + src/vt/cpu/cpu_ops.cpp | 87 ++ src/vt/cuda/cuda_ops.cu | 70 ++ src/vt/op_provider.cpp | 2 + src/vt/ops.cpp | 40 + tests/CMakeLists.txt | 13 + .../test_dflash2_draft_routing.cpp | 93 +- .../vllm/models/test_qwen3_dflash2_draft.cpp | 942 ++++++++++++++++++ .../test_dflash2_selector_refusal.cpp | 105 ++ tests/vt/test_ops_dflash2_grouped_conv.cpp | 385 +++++++ 26 files changed, 2541 insertions(+), 245 deletions(-) create mode 100644 .agents/claims/CLAIM-SPEC-DFLASH2-W2.md create mode 100644 tests/vllm/models/test_qwen3_dflash2_draft.cpp create mode 100644 tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp create mode 100644 tests/vt/test_ops_dflash2_grouped_conv.cpp diff --git a/.agents/claims/CLAIM-SPEC-DFLASH2-W2.md b/.agents/claims/CLAIM-SPEC-DFLASH2-W2.md new file mode 100644 index 000000000..3f60b63e9 --- /dev/null +++ b/.agents/claims/CLAIM-SPEC-DFLASH2-W2.md @@ -0,0 +1,5 @@ +# CLAIM-SPEC-DFLASH2-W2 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-SPEC-DFLASH2-W2` | `SPEC-DFLASH2` (`ACTIVE`), `KERNEL-DFLASH2-GROUPED-CONV` (`ACTIVE`) | Claude Code (opus-5), helper role — fresh implementer working from the committed spec `.agents/specs/dflash2-spec-decode.md` | isolated worktree, CPU only; no GPU, no `nvcc`, no oracle run, no lease, no checkpoint download | `row/SPEC-DFLASH2-W2`, issue [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | Owns ONLY wave W2 of the spec's `## Work breakdown`, plus the spec corrections issue [#1327](https://github.com/mudler/vllm.cpp/issues/1327) names: the `vt::DFlashGroupedConv` op (`OpId::kDFlashGroupedConv`, `DFlashGroupedConvArgs`, the CPU reference and the CUDA mirror) and its kernel-matrix row; the conv weights on `Qwen3DFlashConvWeights` / `Qwen3DFlashLayerWeights` / `Qwen3DFlashWeights` and their load in `LoadQwen3DFlash`; `DflashConvPrepare` / `DflashConvFinish` / `CheckDflashConvBatch` and their call sites in all three `Qwen3DFlashModel` layer bodies; the spec's `## Owed` O3 and O4 repairs in `MakeQwen3DFlashDraftConfig` (the `rope_parameters` and `dflash_config.block_size` fallbacks, the optional `layer_types`, and the `attention_sink_bias` refusal); the narrowing of `RefuseDflash2Draft` into `CheckDflash2DraftArm` and the new `RefuseDflash2CandidateSelector` with its two call sites; the conv's block from the resolved `k` in `LoadDflashDraft`; the three suites `tests/vt/test_ops_dflash2_grouped_conv.cpp`, `tests/vllm/models/test_qwen3_dflash2_draft.cpp` and `tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp`, and the updates to `tests/vllm/entrypoints/test_dflash2_draft_routing.cpp` the narrowing forces; the `SPEC-DFLASH2` engine-matrix row; the `KERNEL` count in `scripts/check-agent-record.py`; and the spec's `## Scope`, `## Upstream chain`, `## Gates`, `## Risks/decisions` D9/D10, `## Owed` and `## Now`. EXCLUDES the candidate selector and its top-k (W3), the speculator and its device path walk (W4), the GGUF drafter ARM (W5), and the G1-G5 run gates (W6). EXCLUDES any parity-pin advance and any DFlash1 behaviour beyond the config-builder fallbacks, which are additive and are asserted to leave every published DFlash1 draft byte-for-byte unchanged | `ACTIVE` | 2026-08-19 — W2 landed red-first; every added behaviour mutation-proven with its compile status and a sha256-verified byte-for-byte restore; two gate weaknesses found BY the mutation pass and repaired before landing; CUDA arm written but never compiled (no `nvcc` on this host) and recorded as spec `## Owed` O6 | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 236c960ac..02cbe621a 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -174,7 +174,7 @@ lifecycle are unchanged. | `SPEC-REJECTION` | Rejection sampler. **I3 verify half LANDED (2026-07-24)**: per-request logits EXPANSION to `1 + k_i` rows (`StepInputs::cu_num_logits` / `num_draft_tokens_per_req` / expanded `logits_indices`) plus the GREEDY rejection sampler — accept a draft iff it equals the target argmax at its own position, emit the target argmax on the FIRST mismatch and stop, emit the bonus argmax when all `k_i` accept, `num_sampled = accepted + 1`, `num_rejected = k_i - accepted` (feeds I2's `num_computed_tokens` rollback and `InputBatch::num_accepted_tokens`). One additive vt op (`kGreedyRejectionSample`) with a CPU reference and a CUDA two-phase mirror of upstream's row-argmax + one-thread-per-request accept walk. DEFAULT-OFF and INERT: with no `SpeculativeConfig` no drafts are ever scheduled, `cu_num_logits` is `arange(num_reqs+1)`, `logits_indices` is the pre-change array and the runner never enters the rejection branch. STOCHASTIC/Gumbel, block verification, `apply_sampling_params` over the expanded batch, and the spec grammar bitmask stay DEFERRED (M-mtp-3). **I5b DRAFTER PREFILL INPUT-PREP LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5B`)**: the draft-token input splice this row's I3 note deferred to I5 — `vllm::v1::prepare_prefill_inputs` + its `SpecPrefillInputs` output struct shift each request's `input_ids` left one within its query span, splice the just-sampled next token (`num_sampled>0 ? last_sampled[idx_mapping[r]] : next_prefill_tokens[...]`) into the freed slot, `query_len -= num_rejected`, and emit last-token index / query_start_loc / seq_lens + CG padding (mirror `speculator.py:469-588`, k=1 early-exit :236-238). A HOST routine in a NEW spec_decode-tree TU (no new CUDA kernel; mirrors the DEVICE-NEUTRAL `prepare_inputs`/`combine_sampled_and_draft_tokens` family — the DGX runner leaf ports the loop to the Triton kernel at I5d), unit-gated `test_prepare_prefill_inputs` 7 cases / 27 assertions RED-first, DEFAULT-OFF INERT (nothing calls it until I5d), additive by construction. Row stays `ACTIVE` — the e2e greedy token gate (M-mtp-1) is owed before `DONE` | T1 | `vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:43,101-160`; `rejection_sampler_utils.py:524,564-585,628,828-841,846-849,863-1125`; `vllm/v1/worker/gpu/model_runner.py:866-898,1065-1077`; `vllm/v1/worker/gpu/input_batch.py:303-397,408-453`; **I5b** `vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:469-588,236-238` | `include/vllm/v1/spec_decode/rejection_sampler.h`; `src/vllm/v1/spec_decode/rejection_sampler.cpp`; `include/vt/ops.h` (`kGreedyRejectionSample`, `vt::GreedyRejectionSample`); `src/vt/cpu/cpu_sample.cpp` (CPU reference); `src/vt/cuda/cuda_sample.cu` (`RejectionRowArgmaxKernel` + `GreedyRejectAcceptKernel`); `src/vt/ops.cpp`; `include/vllm/v1/worker/gpu/prepare_inputs.h` + `src/vllm/v1/worker/gpu/prepare_inputs.cpp` (the expansion); `include/vllm/v1/worker/gpu/runner.h` + `src/vllm/v1/worker/gpu/runner.cpp` (`step_num_logits`, `sample_tokens_with_rejection`); **I5b** `include/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.h` + `src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.cpp` — anchor `include/vllm/v1/spec_decode/rejection_sampler.h:96` | `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp`; `tests/vllm/v1/worker/test_prepare_inputs.cpp` (expansion + no-draft byte-identity); `tests/vt/test_cuda_ops.cpp` (CUDA==CPU bit-exact at vocab 248320); **I5b** `tests/vllm/v1/spec_decode/test_prepare_prefill_inputs.cpp` (7 cases / 27 assertions, RED-first) — anchor `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp:128` | [mtp-spec-decode.md §2.4,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-REJECTION-I3`, `CLAIM-SPEC-MTP-I5B` | | `SPEC-GDN-SEGMENTS` | GDN speculative metadata and slot-snapshot rollback. **I4 LANDED (2026-07-24):** the spec/non-spec metadata split with decode→prefill reclassification (the #34845 case), the `T>1`/`IS_SPEC` GDN recurrence with per-timestep state snapshots, the conv sliding window advancing by the ACCEPTED count, and the k+1 state-slot allocation. DEFAULT-OFF and INERT (`num_spec==0` ⇒ `num_spec_decodes==0`, no shipped kernel branched — both spec kernels are NEW op ids). ROLLBACK PROVEN bit-exact: for every rejection point j the surviving SSM state and conv window are memcmp-identical to running only the accepted prefix through the shipped `vt::GdnDecode`/`CausalConv1dUpdate`, at the real 27B (Hv=48) and 35B (Hv=32) GDN dims on CPU and CUDA. MEASURED state cost: one f32 SSM slot = Hv·Dv·Dk·4B ⇒ 144 MiB/req (27B, 48 layers) / 60 MiB/req (35B, 30 layers) per extra slot; k=1 doubles the GDN SSM state. **I5a GDN LAYER ROUTING WIRED (2026-07-24, `CLAIM-SPEC-MTP-I5A`):** `GdnBlockPaged`'s `num_spec_decodes>0` branch now routes a PURE-spec batch through `vt::CausalConv1dSpecUpdate` + `vt::GdnSpecDecode` (mirror `qwen_gdn_linear_attn.py:1344-1357,1455-1475`), and the runner per-step upload (`StepDevInputs`/`BuildStepDevInputs` + the two decode-graph `Refresh` copies) now carries I4's six spec device tensors, gated by the extended `ValidateGdnAttentionMetadata` spec contract. DEFAULT-OFF INERT (`num_spec_decodes==0` ⇒ stub uploads + the identical non-spec branch). BIT-EXACT vs the I4 ops applied as a token-sequential decode chain, at the real 27B/35B GDN dims, via `GdnBlockPagedForTest` (`tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp`, CPU bit-exact + CUDA on-device); RED-first by a reverted stub (spec recurrence zeroed ⇒ 4/8 fail, maxΔ 1.3-1.6). MIXED spec+non-spec batch refused loudly — lands with I5d's runner loop. Row advances to `ACTIVE`: the M-mtp-1 e2e greedy token gate (verify/propose runner wiring) is owed before `DONE`, and `SPEC-MTP` STAYS `GATING` | T1 | `vllm/v1/attention/backends/gdn_attn.py:189-326,413-462`; `fla/ops/fused_sigmoid_gating.py:66-72,103-116,156-166`; `mamba/ops/causal_conv1d.py:818-1067,1181-1184`; `qwen_gdn_linear_attn.py:1329-1576`; `mamba_utils.py:213-234`; `mamba/abstract.py:55-59` | `include/vllm/v1/attention/backends/gdn_attn.h`; `src/vllm/v1/attention/backends/gdn_attn.cpp`; `include/vt/ops.h` (`kGdnSpecDecode`, `kCausalConv1dSpecUpdate`); `src/vt/ops.cpp`; `src/vt/cpu/cpu_ops.cpp`; `src/vt/cuda/cuda_gdn.cu`; `src/vllm/model_executor/models/qwen3_5_common.{h,cpp}` (`MakeQwen3_5KVCacheSpec`); **I5a:** `src/vllm/model_executor/models/qwen3_5.cpp` (`GdnBlockPaged` spec branch, `StepDevInputs`/`BuildStepDevInputs`, `ValidateGdnAttentionMetadata`), `src/vllm/model_executor/models/qwen3_5_internal.h` (`GdnBlockPagedForTest`) | `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp` (20 cases / 483 assertions incl. the full upstream `GDN_BUILD_TEST_CASES` + default-off byte-identity); `tests/vt/test_ops_gdn.cpp` (reject-at-every-j rollback, CPU + CUDA, real dims); `tests/vllm/models/test_model_registry.cpp` (k+1 slot / widened-conv sizing + `num_spec==0` identity); **I5a** `tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp` (spec-routing bit-exact, RED-first) — anchor `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp:83` | [mtp-spec-decode.md §3,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-GDN-I4`, `CLAIM-SPEC-MTP-I5A` | | `SPEC-DFLASH` | Block-diffusion drafter. **READINESS RE-ASSESSED 2026-07-25 (`CLAIM-SPEC-DFLASH-READINESS`, design-only, DONE) against the LANDED MTP machinery (`SPEC-MTP` I1..I7).** Verdict **GREEN, dispatch-ready, NO hardware/oracle/download blocker** (spec [§0](specs/dflash-spec-decode.md)). Refreshed reuse-vs-new map: DFlash gets FREE from landed MTP — the frozen spec-metadata ABI, the greedy rejection sampler (k-general, I3 tested k∈{1,3}), the GDN spec slot path + rollback + mixed spec/non-spec batch (`GdnBlockPagedMixedSpec`/`IndexSelect`/`IndexCopy`, general `num_spec`), the widened-cache-aware conv ops (I5e), the draft-KV layer pattern (`fa_draft`), the I5d/I7 runner verify/propose loop, and **`num_lookahead_tokens=k+1` ALREADY coded** (`speculative.h:91-108` `use_dflash()`); EXTENDS the single I5d-pre `hidden_tap` seam to multi-tap `[T,H×taps]`; builds NEW the `qwen3_dflash` drafter, the project's FIRST non-causal in-block attention primitive, context-KV precompute, `prepare_dflash_inputs`, and the uniform-1+k FULL CG. **k>1 verdict:** the landed rejection + GDN machinery is MECHANICALLY k-general (no `k==1` hardwiring) — DFlash's k=15 blocks need NO mechanism extension, only exercise/validation at scale (D4) + the k+1-slot memory measurement (~2.3 GiB/req 27B GDN state at block-16, the #1 risk, §5). **Checkpoint-fit:** both z-lab drafts EXIST on HF (27B 1.73 GB / 35B 368 MB bf16, DFlashDraftModel) and FIT the 119 GiB pool trivially (drafts NOT yet on dgx — D0 downloads ≤1.73 GB); the active dgx oracle `vllm-oracle-v0.25.0-stage` CONSTRUCTS DFlash (registry `DFlashDraftModel→qwen3_dflash`, speculator dir present) — soft D0 risk = confirm it SERVES DFlash+NVFP4 on sm_121 (non-causal backend; community `AEON-7/vllm-dflash` container proves the combination runs on GB10). W-plan D0-D6 in the spec. **D0+D1 LANDED 2026-07-26 (`CLAIM-DFLASH-D0D1`) on the ADVANCED pin `555967922`/vLLM 0.26.0.dev0 — `SPEC-DFLASH` → `ACTIVE`.** D0 UNBLOCKED (vllm#40898 resolved under `VLLM_USE_V2_MODEL_RUNNER=1`): the mixed-attn z-lab 27B draft CONSTRUCTS + the drafter is ALIVE (acceptance 2.21/8.80/4.75/4.57 > 1, `num_spec=16`, flashinfer-native fp8-KV, goldens committed); gate FORM measured STRICT MODE-MATCHED (vLLM-ON run-deterministic K>=3 but != vLLM-OFF — the k=16 block verify diverges at bf16 near-ties, so NOT the MTP three-way identity). D1 `DF-AUX-TAPS` DONE: `Qwen3_5AuxTaps` + `ModelForwardInput::aux_tap` route to `Qwen3_5{,Dense}Model::ForwardDeviceMultiTap` capturing `(hidden+res)` at `target_layer_ids` into `[T,H×taps]` (eagle3 `_maybe_add_hidden_state`, aux key L+1); config-gated byte-identical off. Unit gate 598 assertions (independent truncated-model reference, RED-first reversed-concat 384 fail); CUDA 697/697 + compute-sanitizer 0; INERTNESS PROVEN — 27B MTP e2e 9/9 + 27B text SACRED 235/235 byte-identical on the new oracle. **D2 `DF-DRAFT-MODEL` CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D2`, kernel row `KERNEL-ATTN-DFLASH-BLOCK`):** the `qwen3_dflash` draft model (plain 5-layer Qwen3-dense reusing `dense_attn_block.h` ops), the project's FIRST non-causal / bidirectional attention primitive `vt::DFlashBlockAttention` (a SEPARATE op — causal `kAttention`/`kPagedAttention` byte-identical), the fc aux-combine, mask-embed, per-layer SWA/full resolution, and the z-lab loader. CPU gate GREEN (op 12/12 incl. RED non-causal; model forward 95/95 incl. RED full-layer-causal-flip + block isolation + fc RED); existing causal `test_ops_attention` 9/9 + `test_qwen3_forward` 1028 UNCHANGED. **D2 GPU PROMOTION GREEN on dgx (`CLAIM-DFLASH-D2`):** CUDA `-Werror` clean, CUDA==CPU 198412/198412 + compute-sanitizer 0, draft-forward parity vs the REAL vLLM draft (fc rel-L2 0.46%, hidden ≤1.3%, 11 STRICT + 5 near-tie ids), 27B SACRED 235/235 + MTP 9/9 byte-identical — **D2 DONE.** **D3 `DF-DRAFT-KV-PREP` DONE 2026-07-26 (`CLAIM-DFLASH-D3`):** `PrecomputeContextKV` + `PrepareDflashInputs` + `ForwardBlockLogitsWithContext` (reuse the UNCHANGED D2 kernel via [context;block]); GPU numeric-parity `test_qwen3_dflash_kvprep_parity` 61/61 (prepare INTEGER bit-exact vs vLLM's Triton kernel, context-KV K/V rel-L2 0.31%/0.26%, 13 STRICT + 3 near-tie = 16/16), CPU 114/114 RED-proven, inertness 235/235 + 9/9 + D2 37/37 byte-identical. **D4 `DF-ENGINE-INTEGRATION` propose brick + `dflash` config-select CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D4D5`):** `DflashProposeBlock`/`SampleDflashBlockDrafts` (the non-autoregressive whole-block propose composing D3 `ForwardBlockLogitsWithContext` + greedy per-mask argmax, anchor not sampled, `dflash/speculator.py:300-413`) + `ParseSpeculativeConfigJson`/`ResolveDflash` accept `method:"dflash"`. CPU gate `test_dflash_propose` 5/19 GREEN (RED-first anchor-read fails 4/5; brick composes forward+sampler; empty-ctx degenerates to D2; config lookahead k+1). Additive + config-gated ⇒ MTP + non-spec byte-identical BY CONSTRUCTION (`git diff --stat` = new speculator TU + config accept-list + CMake + test, NO runner/model/loader/scheduler edit). **D5 `DF-ENGINE-INTEGRATION` runner-loop LANDED + e2e RUNS on dgx 2026-07-26 (`CLAIM-DFLASH-D5`):** full verify/propose loop wired — loader loads the SEPARATE z-lab draft (`LoadDflashDraft`, host bf16 + target-SHARED bf16 embed/lm_head) via a `--speculative-config` `model` key + `ResolveSpecConfig` dflash branch + `runner.set_dflash_draft`; the verify forward captures the D1 multi-tap (`aux_tap`→`ForwardDeviceMultiTap`) instead of the MTP single tap; `propose_drafts_dflash` ACCUMULATES the per-request combined-feature context (`CombineAuxFeatures(aux_tap)`) across steps and honors the `num_rejected` rollback by appending only the `(T_req−num_rejected)` accepted-prefix features, then runs `DflashProposeBlock` (k=16 GDN-spec exercised first time). **e2e (`test_qwen27_dflash_spec_decode`, 4 prompts×32 tok, our-DFlash-ON vs the committed vLLM-DFlash-ON golden): 2/4 STRICT token-exact (fibonacci, three-laws) + acceptance ~ vLLM on ALL 4 (accepted 19/39/29/25 vs golden 17/39/30/25, deltas +2/0/−1/0 — the MANDATORY dead-drafter-trap condition MET).** The 2 divergences (France tok11 `2972`↔`11751`, 17*23 tok12 `567`↔`488`) are SINGLE bf16 near-tie flips (17*23 RE-CONVERGES after one token = proven near-tie; France cascades from one flip) — the ratified near-tie ROOT the D0 gate-form anticipated, rooted in the D3-documented inline bf16 context-KV recompute envelope (~0.3-1.3% rel-L2), NOT a wiring bug (proven by the 2 exact prompts + near-exact acceptance + a non-trivial shared prefix). Inertness GREEN on this build: SACRED `test_qwen27_paged_engine` 235/235 + MTP `test_qwen27_spec_decode` 9/9 byte-identical; CUDA `-Werror` clean; NO new CUDA kernel (host orchestration reusing D1/D2/D3-sanitized ops). **NOT a clean strict-4/4 pass; STRICT 4/4 token-identity + the speed A/B = D6 (the persistent paged draft-KV bit-matching vLLM's fused context-KV projections + the uniform-1+k FULL CG).** Row STAYS `ACTIVE` (correctness at the ratified near-tie envelope; D6 remains) **D6 2026-07-27 (`CLAIM-DFLASH-D6`) — c1 SPEED A/B DONE + STRICT-irreducibility RCA + CG feasibility (records-only, NO source code):** (1) **c1 speed A/B** (`examples/vllm-bench` at `361189a7`, 8 prose+code prompts×256 tok greedy c1, 2 reps): our DFlash-ON = **2.50x TPOT (40.4 vs 101.2 ms) / 2.48x output-tput (24.4 vs 9.86 tok/s)** over our OFF, acceptance 0.22 (3.56/16), rep-stable <1.5%; `benchmark_binding=true`. vs vLLM-DFlash-ON graphed (same workload): vLLM-DFlash-ON graphed = 28.5 tok/s / 35.1 ms TPOT / acceptance_len 4.30 (same 8 prompts, `VLLM_USE_V2_MODEL_RUNNER=1`, mm-off, gpu_util 0.30), so OURS IS ~14% BELOW vLLM-DFlash-ON on output throughput (24.4 vs 28.5 tok/s) - both ~on-par at spec-OFF (9.86 vs 9.83 tok/s), but vLLM extracts a larger DFlash speedup (2.90x vs our 2.47x) because its draft step is fully device-resident + CUDA-graphed (ours host-orchestrates 13 downloads/step) + slightly higher acceptance (~4.3 vs ~3.6 draft tokens/step). The DONE speed bar (ours >= vLLM) is NOT met; closing it = the device-resident draft rewrite + FULL CG (D6 part 2). (2) **STRICT-4/4 proven bf16-IRREDUCIBLE** — the draft KV cache is bf16 not fp8 (`torch_utils.py:398` `auto`→model dtype; the D0 "fp8-KV" was the backend name, not the KV storage dtype), the D3 golden already compares pre-storage bf16 (residual K 0.31%/V 0.26% = sub-ULP kernel noise), and a fused multi-layer KV GEMM is per-element invariant to our per-layer GEMMs ⇒ bit-exact needs vLLM's exact kernels ⇒ the ratified near-tie gate is the FINAL correctness form (no fused-KV code landed). (3) **FULL CG BLOCKED** on a device-resident draft-path rewrite (the D5 path does 13 device→host downloads/step + host `[context;block]` interleaving) — the remaining throughput-parity increment (the perf form of persistent-paged-KV + the graph). Inertness by construction (the gated binary is the D5 binary; SACRED 235/235 + MTP 9/9 stand). Evidence tool `scripts/spec/vllm_dflash_timing.py`. **D7 2026-07-27 (`CLAIM-DFLASH-D7`) — within-step draft forward made DEVICE-RESIDENT (source-owning): `PrecomputeContextKVDevice` keeps per-layer K/V on device; `ForwardBlockLogitsWithContext` builds [context;block] with `vt::IndexCopy`/`IndexSelect` (removes ~30 D→H `Download`s/step). BIT-IDENTICAL (identity bf16↔f32 round-trips replaced) — e2e `test_qwen27_dflash_spec_decode` 27/27 SAME tokens (2/4 STRICT + 2/4 near-tie, acceptance 19/39/29/25), SACRED 235/235 + MTP 9/9, CUDA `-Werror` clean, compute-sanitizer 0 (198412). But the direct old-vs-new A/B = +2.0% output-tput (IN-NOISE) ⇒ D6's "downloads = the ~14% gap" REFUTED by measurement; ours 19.68 tok/s STILL ~33% BELOW vLLM-DFlash-ON 29.2 tok/s (reconstructed 8-prompt set, more prose-heavy); OFF parity our 9.97 ≥ vLLM 9.66. Residual re-attributed: acceptance (ours 2.49 vs vLLM ~3.13 accepted draft-tok/step, bf16-irreducible) + per-step context-KV RECOMPUTE (O(context²), needs the cross-step persistent paged draft-KV store) + eager-vs-graphed. SPEED BAR NOT met; SPEC-DFLASH stays `ACTIVE`; next = persistent paged draft-KV store → then FULL CG. **D9 2026-07-27 (`CLAIM-DFLASH-D9`) — PERSISTENT PAGED DRAFT-KV LANDED (bit-identical, +22.7% throughput, 0.69×→0.917×); D8 acceptance-ceiling REFUTED; residual = FULL CG ONLY:** `qwen3_dflash.cpp` `AppendContextKVHost` (project ONLY newly-accepted rows → per-layer bf16 K/V, append to `PrecomputedContextKV`) + `ForwardBlockLogitsWithPrecomputedKV` (upload the persistent store, NO re-projection) share the core `ForwardWithCtxKVDev` with the old recompute; `runner.cpp::propose_drafts_dflash` swaps the O(context²) per-step recompute (`dflash_ctx_feats_`) for an append-only per-request `dflash_kv_store_` (rollback=don't-append). NO new CUDA kernel; config-gated. BIT-IDENTICAL: CPU `test_dflash_propose` two new D9 cases = exact float equality vs full recompute; GPU e2e `test_qwen27_dflash_spec_decode` **27/27 SAME tokens** (acceptance 19/39/29/25, same divergences France@11/17×23@12); SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean. **A/B (c1, 8 prose+code×256 tok input-len 512, 2 reps <0.1%, `benchmark_binding=true`):** ours-ON **25.75 tok/s** (was D8 20.99, +22.7%) / 38.40 ms TPOT / acc **3.68/step** vs vLLM-ON graphed **28.09** / 35.60 / acc 3.31 = **0.917×** (~8% below, was 0.69×). **Part 1 same-trajectory:** on the 2 token-identical-trajectory prompts ours per-step acceptance == vLLM's EXACTLY (fibonacci 7.80/7.80, three-laws 3.571/3.571, ratio 1.00) AND on the A/B ours acceptance (3.68) is HIGHER than vLLM's (3.31) ⇒ D8's 0.80–0.85× "bf16 acceptance ceiling" is a trajectory-divergence CONFOUND, REFUTED. Residual (~8%) = eager-vs-graphed ONLY (ours ON/OFF 2.60× vs vLLM 2.91×, OFF at parity, recompute eliminated, acceptance higher) — NOT an irreducible ceiling; the FULL uniform-(1+k) CG (device paged-KV store + paged attn, new-CUDA multi-file) is the SOLE un-landed increment. SPEC-DFLASH stays `ACTIVE` (speed not yet ≥ vLLM; residual isolated to FULL CG). **D12 2026-07-27 (`CLAIM-DFLASH-D12`) — A-wire + Part B LANDED + GPU-gated; Part C (capture) remaining; 0.917×:** A-wire makes the D11 Part-A device store the PRODUCTION path (`runner.{h,cpp}` `dflash_kv_store_`→`shared_ptr`, `MakeDeviceKVStore`/`AppendContextKVDevice`/`ForwardBlockLogitsWithDeviceKV`; GPU-gated e2e `test_qwen27_dflash_spec_decode` 27/27 all-exact acceptance 19/39/29/25 + SACRED 235/235 + MTP 9/9 byte-identical, `-Werror` clean). Part B adds `vt::DFlashPagedBlockAttention` (`OpId::kDFlashPagedBlockAttention`), the capture-safe paged kernel with EVERY metadata input a persistent DEVICE tensor and NO function-local host `cu_seqlens` upload (fixes the `cuda_ops.cu:1277-1280` capture-UAF class), gated CPU==CUDA + cross-check vs materialized `DFlashBlockAttention` `test_ops_dflash_paged_block_attn` 795648/795648 + compute-sanitizer 0. Speed 0.917× (A-wire eager + Part B not yet wired into the forward); `benchmark_binding=false`. Part C (static-shape capture + device mask-scatter + `BeginCapture`/replay + the ≥vLLM c1 A/B) is the SOLE remaining piece; if ours-ON-graphed ≥ vLLM-ON → SPEC-DFLASH DONE. Stays `ACTIVE`. **D13 2026-07-27 (`CLAIM-DFLASH-D13`) — Part C LANDED + GPU-GATED; capture-correctness PROVEN; c1 throughput NEAR-PARITY (ours 0.978x, ~2% below vLLM); gap CLOSED 0.917x→0.978x; STAYS `ACTIVE` (≥vLLM bar not yet met):** single-file additive change (`qwen3_dflash.cpp` +368/-58). (C.1) `DflashDeviceKVStore` → fixed-capacity PAGED cache (per-layer pool `[max_pages,16,Hkv,Dh]` + identity `block_table` + `seq_lens`; append = `vt::IndexCopy` scatter at slot==abs-pos, bit-identical to the D9/D11 store). (C.2) `ForwardPagedBody` runs the (1+k) block through the D12 `vt::DFlashPagedBlockAttention` reading the paged store (no `[context;block]` materialization, no function-local host uploads); runner P==1 propose routes through it, P>1 bit-identical materialized fallback. (C.3) per-request CUDA GRAPH over the paged draft step (warm-in-step repopulates the shared pool free-list right before `BeginCapture` — the fix for a `cudaMalloc`-in-capture `Get` miss from the intervening 27B target forward — then `BeginCapture → ForwardPagedBody → EndCaptureGraph`, replay with growing context entering only via in-place `seq_lens`). **Capture-correctness (MANDATORY): `test_qwen27_dflash_spec_decode` 27/27 with the graph (VT_DFLASH_GRAPH=1) BIT-IDENTICAL to eager (=0)** — same divergence tokens (France@11 got[…2972…], 17×23@12 got[…567…]), same acceptance 19/39/29/25 as D5/D7/D9/D12; graph ENGAGED (5 captures C=2048/5/4/15/6, 32+ replays); the token-diff is the capture-safety proof ([[cudagraph-capture-bakes-stack-addresses]]). **c1 A/B (one flock series, cold rep discarded, 8 prompts×256 tok):** our OFF 10.24 / our ON eager-paged 28.65 (28.69,28.61) / **our ON GRAPHED 28.70 (28.70,28.70), TPOT 34.40** / vLLM-ON graphed steady-state 29.35 (tight 3-rep 29.33/29.37/29.33, TPOT 34.07, acc_len 4.44); D9's 28.09 was a colder cross-session outlier — **NEAR-PARITY: ours 0.978× (~2% below) on the rigorous same-session band** (across sessions ours 28.70 falls inside vLLM's observed 28.09–29.37 range). ON/OFF 2.80× (vLLM ~2.98×), our OFF ≥ vLLM OFF. Per the acceptance rule ("below on any axis = an open gap; near-parity is NOT met"), the ≥vLLM bar is NOT met; STAYS `ACTIVE`. Residual (data-grounded): NOT acceptance (ours realized ~3.68 accepted draft-tok/step > vLLM's 3.44) and NOT launch/graph (both graphed, CG neutral) — per-step COMPUTE (~2% slower target-step); next lever = nsys both draft steps (`--cuda-graph-trace=node`), no premature ceiling. **ATTRIBUTION (supersedes D9):** the CUDA graph is perf-NEUTRAL (+0.3%); the ACTUAL lever was the paged context read (C.1/C.2) removing the D9/D12 per-layer `[context;block]` `IndexCopy` materialization of the whole growing context (25.75 D9 → 28.65 eager-paged, +11%) — the roadmap's "the full CG closes the gap" premise is corrected by measurement. Inertness VERIFIED on the capture binary: SACRED 235/235 + MTP 9/9 byte-identical, CUDA `-Werror` clean, no new kernel (D12 paged kernel already memcheck-0 795648), `check-device-leakage` not increased (paged path REMOVES the materialized-buffer allocs + host uploads). `benchmark_binding=true`. Correctness-complete (ratified near-tie); throughput NEAR-PARITY (0.978×, ~2% residual) ⇒ STAYS `ACTIVE` (the capture-correctness gate is MET; the ≥vLLM speed bar is the sole remaining item, a ~2% per-step-compute residual for an nsys). Anchors: `src/vllm/model_executor/models/qwen3_dflash.cpp` (`DflashDeviceKVStore` paged store, `ForwardPagedBody`, the per-request graph in `ForwardBlockLogitsWithDeviceKV`). **D14 2026-07-27 (`CLAIM-DFLASH-D14`) — SPEED GATE MET → SPEC-DFLASH `DONE`:** an nsys (`--cuda-graph-trace=node`) of the graphed spec-on step attributed the D13 ~2% residual to the from-scratch `DFlashPagedBlockAttentionKernel` draft attention (242.9 ms = 1.8% of GPU time, median ~460 us/call over context C~500-640, vs vLLM's fused flash draft-attn ~0.15%; BOTH engines run identical `cutlass_80_wmma` for the draft bf16 GEMMs, so the GEMMs were NOT the gap). Ported it to a WARP-scoped online-softmax variant `DFlashPagedBlockAttentionWarpKernel` (mirrors the shipped `AttentionWarpKernel`: one warp per (block-query,head), `__shfl_xor` butterfly reduction, register accumulator, NO `__syncthreads` storm; SAME paged/block combined-index read + causal/SWA mask + GQA; default ON, `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel for A/B). Draft attn 242.9 → 77.9 ms (3.1x); our-ON c1 28.60 → 29.32 tok/s (+2.5%). **FINAL same-session 3-rep A/B (8 prompts×256 tok, cold leg discarded): our-ON graphed 29.42/29.27/29.32 (med 29.32) vs vLLM-ON graphed 29.240/29.247/29.233 (med 29.240) — our WORST rep (29.27) > vLLM's BEST (29.247), NON-OVERLAPPING bands, 1.003× ⇒ the ≥vLLM speed gate is MET.** Correctness UNCHANGED (output is exact by spec-decode construction — the target verify is untouched, only which draft proposals are accepted can shift): e2e `test_qwen27_dflash_spec_decode` 27/27 with graph==eager BIT-IDENTICAL, acceptance 19/39/29/25 unchanged (draft accepted 1629 identical warp-vs-block across the whole A/B set), 2/4 STRICT (France@11, 17×23@12 unchanged); CUDA==CPU `test_ops_dflash_paged_block_attn` 795648/795648 (warp within the f32 1e-4 / bf16 3e-2 envelope) + compute-sanitizer 0. Inertness SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean; `check-device-leakage` not increased. `benchmark_binding=true`. Block-diffusion drafting is now correctness-complete (ratified near-tie) AND at/above vLLM throughput — this was the roadmap's FINAL open speed item. Anchors: `src/vt/cuda/cuda_ops.cu` (`DFlashPagedBlockAttentionWarpKernel` + `UseDflashAttnBlockKernel`; the D12 block kernel retained as the `VT_DFLASH_ATTN_BLOCK=1` reference). | T1 | `vllm/v1/worker/gpu/spec_decode/dflash/speculator.py`; `vllm/model_executor/models/qwen3_dflash.py`; `vllm/model_executor/models/interfaces.py:1382` (aux value); `eagle3_utils.py:41-56` (+1 shift) | `include/vllm/model_executor/models/qwen3_5.h` (`Qwen3_5AuxTaps`, `ForwardDeviceMultiTap`); `qwen3_5_dense.h`; `model_registry.h` (`aux_tap`); `src/vllm/model_executor/models/qwen3_5.cpp` (`MaybeCaptureAuxTap`/`ValidateAuxTapLayerIds`/`ForwardDeviceMultiTap`); `qwen3_5_moe.cpp`+`qwen3_5_dense.cpp` (routing); D2/D3 `include/vllm/model_executor/models/qwen3_dflash.h` + `src/vllm/model_executor/models/qwen3_dflash{,_weights}.cpp`; D4 `include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h` + `src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp` (`DflashProposeBlock`/`SampleDflashBlockDrafts`); D5 `src/vllm/entrypoints/model_loader.cpp` (`LoadDflashDraft`/`DflashDraft`) + `include/vllm/entrypoints/model_loader.h`; D5 `src/vllm/v1/worker/gpu/runner.cpp` (`set_dflash_draft`/`propose_drafts_dflash`/aux-tap capture) + `include/vllm/v1/worker/gpu/runner.h`; `src/vllm/config/speculative.cpp` + `include/vllm/config/speculative.h` (`ResolveDflash` + `dflash`/`model` parse); D14 warp kernel [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1433) | `tests/vllm/models/test_qwen27_paged_forward.cpp` (multi-tap 598); `tests/vt/test_ops_dflash_block_attn.cpp`; `tests/vllm/models/test_qwen3_dflash_forward.cpp`; `tests/vllm/v1/spec_decode/test_dflash_kvprep.cpp`; `tests/parity/test_qwen3_dflash_{draft,kvprep}_parity.cpp`; D4 `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (5/19, RED-first); D5 `tests/parity/test_qwen27_dflash_spec_decode.cpp` (e2e 27/27, 2/4 strict + acceptance~vLLM); `scripts/spec/d{0,2,3}_dflash_*.py`; `tests/parity/goldens/dflash_27b{,_draft,_kvprep}/`; D6 `scripts/spec/vllm_dflash_timing.py` (vLLM-DFlash c1 timing); D7 device-resident `src/vllm/model_executor/models/qwen3_dflash.cpp` (`PrecomputeContextKVDevice` + `ForwardBlockLogitsWithContext` via `vt::IndexCopy`/`IndexSelect`); D9 persistent paged draft-KV `qwen3_dflash.{h,cpp}` (`AppendContextKVHost`/`ForwardBlockLogitsWithPrecomputedKV`/`ForwardWithCtxKVDev`/`PrecomputedContextKV`) + `runner.{h,cpp}` (`dflash_kv_store_`/`propose_drafts_dflash`) + `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (2 D9 bit-identity cases); D12 A-wire `runner.{h,cpp}` (device store as production path) + D12 Part B `include/vt/ops.h`/`src/vt/ops.cpp`/`src/vt/cpu/cpu_ops.cpp`/`src/vt/cuda/cuda_ops.cu` (`kDFlashPagedBlockAttention`) + `tests/vt/test_ops_dflash_paged_block_attn.cpp` (CPU==CUDA + cross-check, 795648/795648 + sanitizer-0); D13 `src/vllm/model_executor/models/qwen3_dflash.cpp` (fixed-capacity paged `DflashDeviceKVStore` + `ForwardPagedBody` + the per-request draft-step CUDA graph in `ForwardBlockLogitsWithDeviceKV`); D14 [test_ops_dflash_paged_block_attn](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) + [ledger](parity-ledger.md#L738) | [dflash-spec-decode.md](specs/dflash-spec-decode.md) | `DONE` | `489a7544` | -| `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) — a SECOND DFlash architecture, not a change to DFlash.** Upstream leaves the DFlash draft untouched and adds two mechanisms carried by a new architecture: a GROUPED DYNAMIC DEPTHWISE CONVOLUTION wrapped around each attention and each MLP sublayer (`out[i,c] = sum_t (base[t,c] + delta[i,t,g(c)]) * x[i-t,c]`, taps zeroed across the block boundary) so a proposal position sees the ones before it without another backbone pass; and a CANDIDATE SELECTOR replacing the independent per-slot argmax — keep the target head's top-K per slot, score adjacent transitions ` + unary[c]`, walk the best path from the verified anchor, and at T>0 walk by inverse CDF returning q for the lossless verify. **The published checkpoint is the authority on shapes** (`z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors): DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates. `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **One config rule would land silently wrong**: the checkpoint declares all five layers `sliding_attention` AND `is_causal false`, and our causality resolution mirrors the OLD upstream rule (causal iff SWA), so every layer would run CAUSAL, emit plausible tokens, pass a token gate against our own output, and lose only ACCEPTANCE — which no token gate can see, because the verify is lossless. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. **BEYOND-PIN**: the parity pin `555967922` does not carry the architecture at all; anchors cite the PR head, and this row does NOT advance the pin | T1 | [vllm#52816](https://github.com/vllm-project/vllm/pull/52816) @ head `19c9351904df4c63042671bc67a866ca48dc7d6f` (base `9842d701`, 755+/5-, 11 files) `model_executor/models/qwen3_dflash2.py:1-346` + `v1/worker/gpu/spec_decode/dflash2/speculator.py:1-224` + `registry.py` + `config/vllm.py` (`_is_dflash2_draft`) + the `_dflash_layer_causal` edit in `qwen3_dflash.py`; stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883); tests `tests/v1/spec_decode/test_dflash2.py`, `tests/test_config.py::test_dflash2_draft_forces_v2_model_runner`, the edited `test_dflash_causality.py` | **W1 LANDED (the route and the causality rule; NO DFlash2 mechanism).** (1) `is_causal` PRECEDENCE: `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::ResolveQwen3DFlashAttnModes`, with its coercion helper `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::DeclaredCausal`, resolves an explicit top-level `is_causal` BEFORE `dflash_config.causal` and before the legacy `layer_types == sliding_attention` rule, and `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::MakeQwen3DFlashDraftConfig` (moved out of the loader's anonymous namespace so the key it carries is gateable) copies the key off the draft's own `config.json`. Without both halves the published checkpoint runs every layer CAUSAL. (2) The ROUTE: `include/vllm/config/speculative.h::IsDflash2Draft` plus `src/vllm/entrypoints/model_loader.cpp::ReadDflashDraftArchitectures` + `src/vllm/entrypoints/model_loader.cpp::RefuseDflash2Draft`, called from the dflash branch of `LoadedEngine::ResolveSpecConfig` and from the top of `LoadedEngine::FromModelDir`, ahead of every path, config, tokenizer and weight operation. A DFlash2 checkpoint is REFUSED with both missing mechanisms named rather than drafted through the DFlash1 lane, which would succeed silently: its tensor set is DFlash1's plus the conv and selector tensors. (3) The GGUF axis of BOTH halves, which the architecture string cannot reach: the published `z-lab/Qwen3.8-27B-DFlash2-GGUF` @ `57ab3265` writes `general.architecture = "dflash"`, byte-identical to a DFlash1 drafter, and a GGUF carries no `architectures` array at all. `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp::IsDflash2Gguf` keys on the DFlash2-only metadata (`dflash.selector_rank`, `dflash.selector_top_k`, `dflash.conv_kernel_size`) and feeds the same refusal, and `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp::MakeDflashGgufConfig` carries `dflash.attention.causal`, the GGUF spelling of `is_causal`, into the same `raw` key. The shipped DFlash1 GGUF `muse-glimmer-30b-gguf/dflash-kquant.gguf` carries NEITHER, verified by reading both files on 2026-08-19, so its resolution is unchanged. Reused unchanged when the mechanisms land: `vt::DFlashBlockAttention` (`KERNEL-ATTN-DFLASH-BLOCK`), the DFlash runner/rejection/GDN-rollback lane (`SPEC-DFLASH`), and the loader's target-shared `embed_tokens`+`lm_head` (the `TryLoadBf16` comment on `src/vllm/model_executor/models/qwen3_dflash_weights.cpp` and the two calls it documents in `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash`) which is already what a DFlash2 checkpoint needs. Still owed per `## Port map` (W2-W5): `vt::DFlashGroupedConv`, the selector lattice, a top-k that EMITS pairs (extending the sort-free pivot-bracket search at [cuda_sample.cu:297-506](../src/vt/cuda/cuda_sample.cu#L297) rather than porting FlashInfer's 3380-line radix kernel), the DEVICE path walk, and the GGUF drafter arm | **W1 gated CPU, RED-first.** [test_dflash_causality.cpp](../tests/vllm/v1/spec_decode/test_dflash_causality.cpp) 13 cases / 105 assertions (114 with the real GGUFs) — the port of upstream's `test_dflash_causality.py` branch table as edited by vllm#52816, the precedence no upstream row pins (`is_causal` set together with `dflash_config.causal`), the published DFlash2 shape, and the assertion that the z-lab DFlash1 shape resolves EXACTLY as before. RED at 39 assertions / 11 failed before the rule landed. Three of the cases and 32 of the assertions are the IN-FLOW repair of [#1366](https://github.com/mudler/vllm.cpp/issues/1366), raised by this wave's fresh reviewer against the same function: the two rows of upstream's `_resolve_layer_attention` docstring table that its parametrize list never exercises (`layer_types` absent + `use_swa`, and all-full `layer_types` + `use_swa`, both non-causal upstream and both CAUSAL here), plus the `bool()` coercion that made `"is_causal": 0` fall through in silence while the GGUF arm's `KvI64` already honoured it. RED at 103 assertions / 14 failed before the repair, the 4 `CHECK_FALSE(true)` of the `use_swa` arm among them. BOTH #1366 halves are UNREACHED at this merge commit, exactly as D4 is, and an earlier revision of this row's spec claimed otherwise for the `use_swa` half: `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` is the only published draft of that shape, it declares no `layer_types` and `MakeQwen3DFlashDraftConfig` throws `key 'layer_types' not found` on the absent key, its target `MiMoV2ForCausalLM` is `INVENTORIED` and unimplemented here, and `MakeDflashGgufConfig` can never write `use_swa`. Recorded as the spec's `## Owed` O4, owner W2, with the reason the reachability repair was not attempted in flow. [test_dflash2_draft_routing.cpp](../tests/vllm/entrypoints/test_dflash2_draft_routing.cpp) 11 cases / 30 assertions (33 with the real GGUFs) — REACHABILITY, entered at `LoadedEngine::ResolveSpecConfig` and at `LoadedEngine::FromModelDir` against a nonexistent target directory, which is what proves the refusal precedes all weight I/O. RED at 12 assertions / 2 failed, and the GGUF cases RED again at 24 / 1 before their arm landed. Mutation-proven 2026-08-19 in this worktree, each restored byte-for-byte and each verified by sha256: restoring the `is_sliding` fallback (#1366's defect, 4 failed), accepting only a JSON boolean again (5 failed), dropping the named uncoercible-type refusal (2 failed), dropping it silently instead (1 failed), gating the draft-config carry on `.is_boolean()` again (1 failed), swapping the two explicit arms' precedence (2 failed), removing the top-level arm (7c/48a -> 12 failed), inverting the precedence (2 failed), dropping the `is_causal` carry from the draft-config builder (1 failed), deleting the `ResolveSpecConfig` call site (the reachability mutation, 2 failed), breaking the architecture string (4 failed), deleting the `FromModelDir` early guard (2 failed), dropping the GGUF `attention.causal` read (1 failed), neutralising the GGUF arm of the refusal (1 failed), and breaking the DFlash2-only GGUF keys (1 failed) each turn the focused suite RED. Still owed, all in the spec `## Gates`: conv+lattice vs upstream references at the checkpoint's real shapes; draft-token identity vs vLLM built at `19c93519` under the ratified DFlash near-tie envelope (strict identity is bf16-irreducible, `SPEC-DFLASH` D6); ACCEPTANCE measured SAME-TRAJECTORY, because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound D9 refuted; the GGUF arm with a LOWER bound; and a reachability mutation that deletes the selector's production call site. No speed ratio is claimed until the acceptance gate reads | [dflash2-spec-decode.md](specs/dflash2-spec-decode.md), [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `ACTIVE` | `CLAIM-SPEC-DFLASH2-W1` | +| `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) — a SECOND DFlash architecture, not a change to DFlash.** Upstream leaves the DFlash draft untouched and adds two mechanisms carried by a new architecture: a GROUPED DYNAMIC DEPTHWISE CONVOLUTION wrapped around each attention and each MLP sublayer (`out[i,c] = sum_t (base[t,c] + delta[i,t,g(c)]) * x[i-t,c]`, taps zeroed across the block boundary) so a proposal position sees the ones before it without another backbone pass; and a CANDIDATE SELECTOR replacing the independent per-slot argmax — keep the target head's top-K per slot, score adjacent transitions ` + unary[c]`, walk the best path from the verified anchor, and at T>0 walk by inverse CDF returning q for the lossless verify. **The published checkpoint is the authority on shapes** (`z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors): DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates. `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **One config rule would land silently wrong**: the checkpoint declares all five layers `sliding_attention` AND `is_causal false`, and our causality resolution mirrors the OLD upstream rule (causal iff SWA), so every layer would run CAUSAL, emit plausible tokens, pass a token gate against our own output, and lose only ACCEPTANCE — which no token gate can see, because the verify is lossless. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. **BEYOND-PIN**: the parity pin `555967922` does not carry the architecture at all; anchors cite the PR head, and this row does NOT advance the pin | T1 | [vllm#52816](https://github.com/vllm-project/vllm/pull/52816) @ head `19c9351904df4c63042671bc67a866ca48dc7d6f` (base `9842d701`, 755+/5-, 11 files) `model_executor/models/qwen3_dflash2.py:1-346` + `v1/worker/gpu/spec_decode/dflash2/speculator.py:1-224` + `registry.py` + `config/vllm.py` (`_is_dflash2_draft`) + the `_dflash_layer_causal` edit in `qwen3_dflash.py`; stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883); tests `tests/v1/spec_decode/test_dflash2.py`, `tests/test_config.py::test_dflash2_draft_forces_v2_model_runner`, the edited `test_dflash_causality.py` | **W1 LANDED (the route and the causality rule; NO DFlash2 mechanism).** (1) `is_causal` PRECEDENCE: `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::ResolveQwen3DFlashAttnModes`, with its coercion helper `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::DeclaredCausal`, resolves an explicit top-level `is_causal` BEFORE `dflash_config.causal` and before the legacy `layer_types == sliding_attention` rule, and `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::MakeQwen3DFlashDraftConfig` (moved out of the loader's anonymous namespace so the key it carries is gateable) copies the key off the draft's own `config.json`. Without both halves the published checkpoint runs every layer CAUSAL. (2) The ROUTE: `include/vllm/config/speculative.h::IsDflash2Draft` plus `src/vllm/entrypoints/model_loader.cpp::ReadDflashDraftArchitectures` + `src/vllm/entrypoints/model_loader.cpp::CheckDflash2DraftArm`, called from the dflash branch of `LoadedEngine::ResolveSpecConfig` and from the top of `LoadedEngine::FromModelDir`, ahead of every path, config, tokenizer and weight operation. A DFlash2 checkpoint is REFUSED with both missing mechanisms named rather than drafted through the DFlash1 lane, which would succeed silently: its tensor set is DFlash1's plus the conv and selector tensors. (3) The GGUF axis of BOTH halves, which the architecture string cannot reach: the published `z-lab/Qwen3.8-27B-DFlash2-GGUF` @ `57ab3265` writes `general.architecture = "dflash"`, byte-identical to a DFlash1 drafter, and a GGUF carries no `architectures` array at all. `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp::IsDflash2Gguf` keys on the DFlash2-only metadata (`dflash.selector_rank`, `dflash.selector_top_k`, `dflash.conv_kernel_size`) and feeds the same refusal, and `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp::MakeDflashGgufConfig` carries `dflash.attention.causal`, the GGUF spelling of `is_causal`, into the same `raw` key. The shipped DFlash1 GGUF `muse-glimmer-30b-gguf/dflash-kquant.gguf` carries NEITHER, verified by reading both files on 2026-08-19, so its resolution is unchanged. Reused unchanged when the mechanisms land: `vt::DFlashBlockAttention` (`KERNEL-ATTN-DFLASH-BLOCK`), the DFlash runner/rejection/GDN-rollback lane (`SPEC-DFLASH`), and the loader's target-shared `embed_tokens`+`lm_head` (the `TryLoadBf16` comment on `src/vllm/model_executor/models/qwen3_dflash_weights.cpp` and the two calls it documents in `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash`) which is already what a DFlash2 checkpoint needs. **W2 LANDED (the grouped dynamic depthwise convolution, REACHED).** `vt::DFlashGroupedConv` (`OpId::kDFlashGroupedConv`, kernel-matrix row `KERNEL-DFLASH2-GROUPED-CONV`) is the project's FIRST dynamic, grouped, block-masked convolution: `out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, tap `t` live only where `(i mod (1+k)) >= t`, `g(c) = c / conv_group_size`, and `base_kernel` dim 0 the prepare/finish SIDE rather than a tap. `src/vt/cpu/cpu_ops.cpp::DFlashGroupedConvKernel` is the authoritative reference and rounds to the tensor dtype after each step, mirroring upstream's bf16 chain, so `src/vt/cuda/cuda_ops.cu::DFlashGroupedConvKernelCuda` is specified BIT-IDENTICAL rather than within an envelope. Both of upstream's position-mask arms are ported (`pos & (block-1)` and `pos % block`) and gated at block 5, 8 and 16. The conv weights load in `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash` under the published tensor names, and `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvPrepare` / `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvFinish` run them in ALL THREE draft layer bodies, the third being the paged body the production decode path reaches through `ForwardBlockLogitsWithDeviceKV`. **The REFUSAL moved so the conv could be reached** (spec D10): a safetensors DFlash2 draft is now ADMITTED at `src/vllm/entrypoints/model_loader.cpp::CheckDflash2DraftArm` with a startup NOTICE naming the boundary, and refused BY NAME at `src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp::RefuseDflash2CandidateSelector` AFTER the block forward; the GGUF arm keeps its startup refusal and moves with W5. W2 also discharged the spec's `## Owed` O3 and O4: `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::MakeQwen3DFlashDraftConfig` could not parse EITHER published DFlash2 `config.json` (`rope_theta` and `block_size` are nested under `rope_parameters` and `dflash_config`) nor MiMo's DFlash1 draft (no `layer_types`), and `dflash_config.attention_sink_bias` is now REFUSED BY NAME because this lane has no attention sink and the parse repair alone would have turned a loud error into a quiet wrong answer. Still owed per `## Port map` (W3-W5): the selector lattice, a top-k that EMITS pairs (extending the sort-free pivot-bracket search at [cuda_sample.cu:297-506](../src/vt/cuda/cuda_sample.cu#L297) rather than porting FlashInfer's 3380-line radix kernel), the DEVICE path walk, and the GGUF drafter arm | **W1 gated CPU, RED-first.** [test_dflash_causality.cpp](../tests/vllm/v1/spec_decode/test_dflash_causality.cpp) 13 cases / 105 assertions (114 with the real GGUFs) — the port of upstream's `test_dflash_causality.py` branch table as edited by vllm#52816, the precedence no upstream row pins (`is_causal` set together with `dflash_config.causal`), the published DFlash2 shape, and the assertion that the z-lab DFlash1 shape resolves EXACTLY as before. RED at 39 assertions / 11 failed before the rule landed. Three of the cases and 32 of the assertions are the IN-FLOW repair of [#1366](https://github.com/mudler/vllm.cpp/issues/1366), raised by this wave's fresh reviewer against the same function: the two rows of upstream's `_resolve_layer_attention` docstring table that its parametrize list never exercises (`layer_types` absent + `use_swa`, and all-full `layer_types` + `use_swa`, both non-causal upstream and both CAUSAL here), plus the `bool()` coercion that made `"is_causal": 0` fall through in silence while the GGUF arm's `KvI64` already honoured it. RED at 103 assertions / 14 failed before the repair, the 4 `CHECK_FALSE(true)` of the `use_swa` arm among them. BOTH #1366 halves are UNREACHED at this merge commit, exactly as D4 is, and an earlier revision of this row's spec claimed otherwise for the `use_swa` half: `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` is the only published draft of that shape, it declares no `layer_types` and `MakeQwen3DFlashDraftConfig` throws `key 'layer_types' not found` on the absent key, its target `MiMoV2ForCausalLM` is `INVENTORIED` and unimplemented here, and `MakeDflashGgufConfig` can never write `use_swa`. Recorded as the spec's `## Owed` O4, owner W2, with the reason the reachability repair was not attempted in flow. [test_dflash2_draft_routing.cpp](../tests/vllm/entrypoints/test_dflash2_draft_routing.cpp) 11 cases / 30 assertions (33 with the real GGUFs) — REACHABILITY, entered at `LoadedEngine::ResolveSpecConfig` and at `LoadedEngine::FromModelDir` against a nonexistent target directory, which is what proves the refusal precedes all weight I/O. RED at 12 assertions / 2 failed, and the GGUF cases RED again at 24 / 1 before their arm landed. Mutation-proven 2026-08-19 in this worktree, each restored byte-for-byte and each verified by sha256: restoring the `is_sliding` fallback (#1366's defect, 4 failed), accepting only a JSON boolean again (5 failed), dropping the named uncoercible-type refusal (2 failed), dropping it silently instead (1 failed), gating the draft-config carry on `.is_boolean()` again (1 failed), swapping the two explicit arms' precedence (2 failed), removing the top-level arm (7c/48a -> 12 failed), inverting the precedence (2 failed), dropping the `is_causal` carry from the draft-config builder (1 failed), deleting the `ResolveSpecConfig` call site (the reachability mutation, 2 failed), breaking the architecture string (4 failed), deleting the `FromModelDir` early guard (2 failed), dropping the GGUF `attention.causal` read (1 failed), neutralising the GGUF arm of the refusal (1 failed), and breaking the DFlash2-only GGUF keys (1 failed) each turn the focused suite RED. **W2 gated CPU.** [test_ops_dflash2_grouped_conv.cpp](../tests/vt/test_ops_dflash2_grouped_conv.cpp) 6 cases / **9410 assertions**, `Status: SUCCESS!`, exit 0 -- upstream's own sequential reference loop from `tests/v1/spec_decode/test_dflash2.py` at block 5 (the `% block` arm), 8 and 16 (the two PUBLISHED checkpoints; upstream's parametrize covers 5 and 8 only, so 16 is ours), both published taps/group shapes on both sides, plus hand-computed corners for the block boundary, the group map and the side. [test_qwen3_dflash2_draft.cpp](../tests/vllm/models/test_qwen3_dflash2_draft.cpp) 16 cases / 108 assertions, `Status: SUCCESS!`, exit 0 -- the two PUBLISHED DFlash2 `config.json` documents verbatim (sha256 recorded) through the production builder, MiMo's `dflash/config.json` verbatim, conv weights read off a REAL on-disk safetensors shard by the production loader, an IDENTITY conv proven BIT-IDENTICAL to no conv, and each conv driven ALONE through each layer body. [test_dflash2_selector_refusal.cpp](../tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp) 3 cases / 14 assertions. RED-first at 5 cases / 4 failed with `[json.exception.out_of_range.403] key 'rope_theta' not found`, which is O3 exactly as the spec predicted it. Mutation-proven 2026-08-19, each restored byte-for-byte and verified by sha256, each with its compile status printed: deleting the conv call sites in `ForwardBlockLogits` (5 cases / 9 assertions red), in `ForwardWithCtxKVDev` (1/1) and in `ForwardPagedBody` (1/1); forcing `args.side` to 0 (op 2/4353, model 1/1); dropping the block mask (3/449); the wrong group map (3/7436); dropping the `rope_parameters` fallback (2/2), the `dflash_config.block_size` fallback (2/2), the `layer_types` fallback (2/2), the `attention_sink_bias` refusal (1/1), the uniform-block guard (1/1) and the `DflashProposeBlock` selector-refusal call (1/1); and restoring W1's startup refusal (3 cases / 2 assertions red). TWO GATE WEAKNESSES were found BY that pass and repaired before landing rather than hidden: activating both convs at once could not see one missing call site, and the first side probe could not see a forced side. Three mutations came back GREEN and are recorded as `## Owed` rather than as passes: the loader's own `conv_block_size = k + 1` (O5), the CUDA arm which has NEVER COMPILED on this host (O6), and the runner's selector-refusal call site (O7). Still owed, all in the spec `## Gates`: the lattice half of G1, and the conv's CUDA arm; conv+lattice vs upstream references at the checkpoint's real shapes; draft-token identity vs vLLM built at `19c93519` under the ratified DFlash near-tie envelope (strict identity is bf16-irreducible, `SPEC-DFLASH` D6); ACCEPTANCE measured SAME-TRAJECTORY, because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound D9 refuted; the GGUF arm with a LOWER bound; and a reachability mutation that deletes the selector's production call site. No speed ratio is claimed until the acceptance gate reads | [dflash2-spec-decode.md](specs/dflash2-spec-decode.md), [#1314](https://github.com/mudler/vllm.cpp/issues/1314), [#1327](https://github.com/mudler/vllm.cpp/issues/1327) | `ACTIVE` | `CLAIM-SPEC-DFLASH2-W2` | | `SPEC-DSPARK` | DSpark semi-autoregressive block drafter for DeepSeek-V4 and Qwen3, including native and Speculators checkpoint layouts, anchor-vs-bonus-token semantics, reduced/heterogeneous vocabulary mapping, sequential Markov sampling, noncausal draft attention, rejection/metrics and full-CUDA-graph compatibility; user-promoted scope at the v0.25.0 audit; **re-grounded at the CURRENT pin 2026-08-08 (USER-requested rider, task #287):** `DSparkSpeculator(DFlashSpeculator)` drafts a block in ONE parallel pass (anchor + N−1 noise queries); method `"dspark"` V2-runner-ONLY (forced at `config/vllm.py:560-568`, "parallel drafting natively in V2" `:2173-2177`); draft models exist for BOTH our registered target families (Qwen3 + Gemma4) and slot beside our landed MTP + DFlash lanes (`src/vllm/v1/worker/gpu/spec_decode/{mtp,dflash}/speculator.cpp`). **SPIKE COMMITTED 2026-08-09 (`CLAIM-SPEC-DSPARK`, developer goal "a full DSpark implementation, based on vLLM"):** the whole upstream surface is 1613 lines over 5 files, 3 of them `class X(DFlashY)` subclasses; the delta over our landed `SPEC-DFLASH` lane is A) the low-rank Markov logit-bias head (`markov_rank=256` in every shipped ckpt), B) the sequential N-step sample loop, C) the `sample_from_anchor` N-query layout (the `PrepareDflashInputs` field already exists, always `false` today; `NumLookaheadTokens()` already returns `k` for dspark — verified), D) reduced draft vocab + `d2t`, E) config/method resolution incl. the `k >= dspark_block_size` hard error, F) Speculators-format config translation (a subsystem we have ZERO of today). Gate-model drafts EXIST (`RedHatAI/Qwen3.6-35B-A3B-speculator.dspark` 1.90 GB, `satgeze/Qwen3.6-27B-DSpark` 8.80 GB) and the upstream test's own 4B pair (`Qwen/Qwen3-4B-FP8` + `deepseek-ai/dspark_qwen3_4b_block7`, 2.79 GB) is the smallest honest lane; DeepSeek-V4 DSpark stays OUT OF SCOPE (HW-blocked) | T1 | `vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:37`; `vllm/config/speculative.py:62,310,706-709`; `vllm/model_executor/models/qwen3_dspark.py:36,95` + `gemma4_dspark.py:134,182`; registry `registry.py:609,611` @ `555967922` | - | - | [dspark-spec-decode.md](specs/dspark-spec-decode.md) (**spike, 2026-08-09**; supersedes the [grounding note](specs/dspark-speculator-note.md)) | `ACTIVE` | `CLAIM-SPEC-DSPARK` | | `SPEC-TLI` | Tokenizer-agnostic speculative decoding across heterogeneous draft/target vocabularies: shared-token mapping, target↔draft ID translation, constrained draft logits and greedy-only validation | T1 | `vllm/config/speculative.py:145-149,1173-1203`; `vllm/v1/spec_decode/vocab_mapping.py:68-160`; `vllm/v1/spec_decode/draft_model.py:34-58`; `vllm/v1/spec_decode/llm_base_proposer.py:432-495,688-691,831-837`; `tests/v1/spec_decode/test_vocab_mapping.py:1-50` @ `702f481` | - | - | `planned: specs/tli-spec-decode.md` | `INVENTORIED` | - | | `SPEC-NGRAM` | Draft-FREE n-gram proposer. **DONE 2026-07-27 (`CLAIM-ROADMAP-D3`):** 1:1 port of `ngram_proposer.py` (KMP-LPS suffix-ngram matcher + batch propose) wired as a third `--speculative-config` method reusing the LANDED MTP/DFlash verify/reject/`take_draft_token_ids` loop (no draft model / hidden tap / draft KV; GDN spec verify reused via `MakeQwen3_5KVCacheSpec(num_spec>0)`). 27B gate 5/5 STRICT our-ngram-ON == vLLM-ngram-ON + 180/180 drafts accepted; unit 19/19; spec-OFF byte-identical; host-side, no new kernel, `-Werror` clean | T2 | `vllm/v1/spec_decode/ngram_proposer.py:184-276,128-180`; `vllm/config/speculative.py:734-762,1224-1234`; `tests/v1/spec_decode/test_ngram.py` @ `555967922` | `src/vllm/v1/spec_decode/ngram_proposer.{h,cpp}`; `include/vllm/config/speculative.h` (`ResolveNgram`/`use_ngram`); `src/vllm/config/speculative.cpp`; `src/vllm/entrypoints/model_loader.cpp` (`ResolveSpecConfig`); `src/vllm/v1/worker/gpu/runner.cpp` (`propose_drafts_ngram`) | `tests/vllm/v1/spec_decode/test_ngram_proposer.cpp` (19/19); `tests/parity/test_qwen27_ngram_spec_decode.cpp` (5/5 STRICT, 180/180 accepted, dgx); golden `tests/parity/goldens/ngram_27b/ngram_27b_spec_on.json` + `scripts/spec/ngram_27b_golden.py`; ledger [parity-ledger.md](parity-ledger.md) 2026-07-27 — anchor `tests/vllm/v1/spec_decode/test_ngram_proposer.cpp:30` | [specs/spec-decode-breadth-d3.md](specs/spec-decode-breadth-d3.md) | `ACTIVE` | `CLAIM-ROADMAP-D3` | @@ -224,7 +224,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1237` (the bare form), `cell_citations` `scripts/check-agent-record.py:1291` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1348` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1235` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1471` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fe4b5e576..090cf4113 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -457,3 +457,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug | | [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement | | [#1386](https://github.com/mudler/vllm.cpp/issues/1386) | — | `tools/bench/gpu_clock_state.py`'s `QUERY_FIELDS` collects nine fields and **none of them is thermal and none is electrical**, so the driver's own `SwThermalSlowdown` label can never be checked against a die reading on any window this helper has ever recorded. The measured consequence is that the nine windows of 2026-08-19 cannot distinguish a load transition from a thermal excursion. The concrete evidence is `clock-c1-r1.samples.json` in `/mnt/nas_share/rc/q38bf16/out/bench-20260819T035148Z/`: ours c1 r1 dips five times on the same period at the same `utilization.gpu = 96` — 48.83 s / 2177 MHz, 80.60 s / 2320 MHz, 109.28 s / 2210 MHz, 137.98 s / 2359 MHz, 166.07 s / 2268 MHz — and **two of those five carry `0x0000000000000000`**, no throttle bit at all (2210 and 2359), while three carry `0x20`. The 2210 MHz unlabelled dip is deeper than two of the three labelled ones, so the driver labels comparable excursions inconsistently and the bit alone cannot decide it. What would settle it: add `temperature.gpu` and `power.draw` to `QUERY_FIELDS`. That changes the clock-record schema, so it owes its own row and spec. Split out of [#1354](https://github.com/mudler/vllm.cpp/issues/1354) and owed under `## Owed` in [lease-clock-pinning.md](specs/lease-clock-pinning.md) | gap | +| [#1327](https://github.com/mudler/vllm.cpp/issues/1327) | `SPEC-DFLASH2` | `.agents/specs/dflash2-spec-decode.md` `## Upstream chain` said the three output scalars `input_embedding_scale`, `output_multiplier` and `final_logit_softcapping` are "ABSENT from this config" and that "no published checkpoint exercises them, so the port implements them and gates them synthetically". That was measured on `z-lab/Qwen3.8-27B-DFlash2` alone. `z-lab/Muse-Glimmer-30B-DFlash2` — the SECOND published DFlash2 checkpoint, `config.json` sha256 `cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9`, read 2026-08-19 — sets `output_multiplier` to `0.19611613513818404` and `final_logit_softcapping` to `20.0`, and ships `block_size` 16 against the 27B's 8, hidden 6656 (416 groups, a 1664-wide `kernel_projection`) and `rope_theta` 500000.0. Both scalars are applied to the candidate VALUES in `compute_candidates` BEFORE the selector scores them, so a wrong value reorders the top-K and moves acceptance without raising — the `is_causal` failure class one layer up, and the class no token gate here can see. A port reading all three with `.get(key, default)` would pass every gate built from the 27B draft and be measuring the default path. The same file also falsifies `## Scope`'s exclusion of "a second DFlash2 target family": upstream registers ONE architecture and both checkpoints declare `model_type` `qwen3`, so what the second adds is values rather than a class. FIXED IN FLOW by `SPEC-DFLASH2` W2, which is the wave that had to read both configs anyway: `## Scope` drops the exclusion, `## Upstream chain` records both values with their source, `## Gates` G1 now requires BOTH published block shapes (upstream's own reference test parametrises 5 and 8 and never reaches 16), and `## Risks/decisions` D9 binds W3 to gate the scalars against the checkpoint that sets them | gap | diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index c4e96c5a5..7688cab63 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -140,6 +140,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:509](../include/vllm/v1/attention/backend.h#L509), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | | `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:1843](../src/vt/cpu/cpu_ops.cpp#L1843) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:1300](../src/vt/cuda/cuda_ops.cu#L1300) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | | `KERNEL-ATTN-DFLASH-PAGED-BLOCK` | **DFlash PAGED in-block attention — the CAPTURE-SAFE form of `KERNEL-ATTN-DFLASH-BLOCK`** (SPEC-DFLASH D12 Part B, the CUDA-graph draft-attention primitive). The (1+k) block queries attend over `[PAGED context ; their own (1+k) block]`: the growing context enters as DATA (paged K/V cache `[pages,block_size,Hkv,D]` + per-request `seq_lens` + `block_table`, mirroring `PagedAttentionKernel`) instead of a variable-size materialized combined buffer, so the launch grid is STATIC over the fixed `Nq=(1+k)*num_reqs` rows and EVERY metadata input is a persistent DEVICE tensor read in place — NO `cudaMallocAsync`/`cudaMemcpyAsync` of a function-local host `cu_seqlens` (the [[cudagraph-capture-bakes-stack-addresses]] UAF class the eager `LaunchDFlashBlockAttention` had). Same f32 online softmax + D2 in-block mask over the COMBINED index; bit-identical to `DFlashBlockAttention` over the materialized `[context;block]` buffer | vLLM full CG `dflash/cudagraph.py` + `speculator.py:411-458` + `precompute_and_store_context_kv` (`qwen3_dflash.py:548-619`) @ `555967922`; paged read mirrors our `PagedAttentionKernel` [cuda_paged_attn.cu:184](../src/vt/cuda/cuda_paged_attn.cu#L184) | `OpId::kDFlashPagedBlockAttention` + `DFlashPagedBlockAttentionArgs` + decl [ops.h](../include/vt/ops.h) + wrapper/validation [ops.cpp](../src/vt/ops.cpp); CPU REFERENCE `DFlashPagedBlockAttentionKernel` [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp); CUDA `DFlashPagedBlockAttentionKernelCuda` [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1452) (static grid, persistent device metadata) + D14 WARP variant [DFlashPagedBlockAttentionWarpKernel](../src/vt/cuda/cuda_ops.cu#L1433) | **GPU GATE GREEN on dgx (2026-07-27, GB10 sm_121a):** CUDA `-Werror` clean (0 warnings); [test_ops_dflash_paged_block_attn.cpp](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) cross-checks CPU-paged == materialized `DFlashBlockAttention` across 6 corners (non-causal, causal-SWA, block isolation, GQA, multi-page, zero-context) + CUDA==CPU (f32+bf16) = **795648/795648 assertions**; `compute-sanitizer --tool memcheck` **0 errors**. **D13 (2026-07-27, `CLAIM-DFLASH-D13`): WIRED INTO PRODUCTION** — the single-request DFlash draft forward (`ForwardPagedBody`, `qwen3_dflash.cpp`) now runs the (1+k) block through this kernel reading a fixed-capacity paged `DflashDeviceKVStore`, and the whole draft step is captured into a per-request CUDA graph + replayed (the growing context enters only via the in-place `seq_lens`). Capture-correctness PROVEN: `test_qwen27_dflash_spec_decode` 27/27 with the graph BIT-IDENTICAL to eager (same tokens + acceptance 19/39/29/25); c1 throughput NEAR-PARITY with vLLM-DFlash-ON (ours 0.978×, ~2% below the tight 3-rep band; gap closed 0.917×→0.978× via the paged read, the CG is perf-neutral) — the kernel is landed + wired + gated; STAYS `ACTIVE` with the engine feature (the ~2% ≥vLLM residual is per-step compute for an nsys). **D14 (2026-07-27, `CLAIM-DFLASH-D14`): the residual WAS this kernel → WARP-scoped variant added → SPEED GATE MET, `DONE`.** An nsys (`--cuda-graph-trace=node`) attributed the D13 ~2% residual to THIS kernel: `DFlashPagedBlockAttentionKernel` = 242.9 ms = 1.8% of the graphed step's GPU time, median ~460 us/call (grid `(nq=17,hq=32)` × kBlock=256 threads looping SERIALLY over C~500-640 keys with a 256-wide shared-mem tree reduction + 2 `__syncthreads` PER key — the latency/sync storm the ViT tower fixed with `AttentionDenseFast`), vs vLLM's fused flash draft-attn ~0.15%. Added `DFlashPagedBlockAttentionWarpKernel` ([cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)): ONE WARP per (block-query,head), `__shfl_xor` butterfly head_dim reduction, register accumulator, NO `__syncthreads`; SAME paged/block combined-index read + causal/SWA mask + GQA (copied verbatim from the block kernel), mirroring the shipped `AttentionWarpKernel`. Default ON; `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel. Draft attn **242.9 → 77.9 ms (3.1×)**; our-ON c1 **28.60 → 29.32 tok/s**; FINAL 3-rep A/B our-ON 29.32 ≥ vLLM-ON 29.240 (non-overlapping bands, 1.003×) ⇒ **≥vLLM MET**. Not bit-identical to the block kernel but same f32-online-softmax math within envelope; CUDA==CPU `test_ops_dflash_paged_block_attn` **795648/795648** (f32 1e-4/bf16 3e-2) + **compute-sanitizer 0**; e2e 27/27 graph==eager, acceptance 19/39/29/25 unchanged (1629 accepted identical warp-vs-block); SACRED 235/235 + MTP 9/9 inert; `-Werror` clean; closure [ledger](parity-ledger.md#L738) | [DFlash spec](specs/dflash-spec-decode.md) §0 D12/D13/D14 | `DONE` | `489a7544` | +| `KERNEL-DFLASH2-GROUPED-CONV` | **DFlash2 grouped dynamic depthwise convolution — the project's FIRST dynamic (input-conditioned) convolution kernel** (SPEC-DFLASH2 W2, #1314). `out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, with tap `t` contributing only where `(i mod block_size) >= t`, `g(c) = c / group_size`, and `block_size` the QUERY block `1 + k`. Three things separate it from the shipped `KERNEL-DEPTHWISE-CONV1D`: the kernel is DYNAMIC (a per-position `delta` projected from the sublayer input, added to a static per-channel `base`), it is GROUPED (one delta per group of channels, one base per channel), and its taps are ZEROED ACROSS THE BLOCK BOUNDARY rather than across the sequence — which is what lets a proposal position see the ones before it without another backbone pass. `base_kernel` dim 0 is the prepare/finish SIDE and not a tap; on the published 27B draft both axes are 2, so nothing but the port note and the shape assertion separates a correct load from a transposed one. Every intermediate rounds to the tensor dtype, mirroring upstream's bf16 chain, so the op is elementwise with NO reduction-order freedom and the CUDA arm is specified BIT-IDENTICAL to CPU rather than within an envelope | **BEYOND-PIN** — `vllm/model_executor/models/qwen3_dflash2.py` (`_grouped_conv`, `DFlashGroupedConv`, `DFlash2Qwen3DecoderLayer.forward`) @ [vllm-project/vllm#52816](https://github.com/vllm-project/vllm/pull/52816) head `19c9351904df4c63042671bc67a866ca48dc7d6f`; the parity pin `555967922` does not carry the architecture and this row does NOT advance it | `OpId::kDFlashGroupedConv` + `DFlashGroupedConvArgs` + decl/wrapper `include/vt/ops.h::DFlashGroupedConv` and `src/vt/ops.cpp::DFlashGroupedConv`; CPU REFERENCE `src/vt/cpu/cpu_ops.cpp::DFlashGroupedConvKernel` (the authoritative impl); CUDA mirror `src/vt/cuda/cuda_ops.cu::DFlashGroupedConvKernelCuda` (one thread per (row, channel); `__fadd_rn`/`__fmul_rn` forbid the FMA contraction the CPU build pins off). Consumed by the draft through `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvPrepare` and `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvFinish`, called from all THREE layer bodies, with the uniform-block precondition in `src/vllm/model_executor/models/qwen3_dflash.cpp::CheckDflashConvBatch`; weights loaded by `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash` | **CPU GATE GREEN 2026-08-19** ([test_ops_dflash2_grouped_conv.cpp](../tests/vt/test_ops_dflash2_grouped_conv.cpp)) 6 cases / **9410 assertions**, `Status: SUCCESS!`, exit 0 — upstream's own sequential reference loop at block 5 (the `% block` arm), 8 and 16 (the two PUBLISHED checkpoints, `z-lab/Qwen3.8-27B-DFlash2` and `z-lab/Muse-Glimmer-30B-DFlash2`; upstream's parametrize covers 5 and 8 only), both published taps/group shapes on both sides, plus hand-computed corners for the block boundary, the group map and the side. MODEL GATE GREEN ([test_qwen3_dflash2_draft.cpp](../tests/vllm/models/test_qwen3_dflash2_draft.cpp)) 16 cases / 108 assertions, `Status: SUCCESS!`, exit 0 — weights read off a REAL on-disk safetensors shard by the production loader, an IDENTITY conv proven BIT-IDENTICAL to no conv, and each conv driven ALONE through each of the three layer bodies. MUTATION-PROVEN 2026-08-19, each restored byte-for-byte and verified by sha256: deleting the call sites in `ForwardBlockLogits` (5 cases / 9 assertions red), in `ForwardWithCtxKVDev` (1/1) and in `ForwardPagedBody` (1/1); forcing `args.side` to 0 (op 2 cases/4353 assertions red, model 1/1); dropping the block mask (3/449); the wrong group map (3/7436); and dropping the uniform-block guard (1/1). TWO gate repairs came out of that pass and are recorded rather than hidden: activating both convs at once could not see one missing call site, and the first side probe could not see a forced side. **CUDA UNVERIFIED and OWED** — the kernel and its registration are written and the CUDA==CPU bit-identity case exists over six shapes, but the authoring host has no `nvcc`, so it has NEVER COMPILED and the case reports `no CUDA backend; skipping`. Spec `## Owed` O6, owed to the operator's GPU lease | [DFlash2 spec](specs/dflash2-spec-decode.md) W2, [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `ACTIVE` | `CLAIM-SPEC-DFLASH2-W2` | | `KERNEL-ATTN-DSA-SPARSE-INDEX` | **DeepSeek-V4 DSA "Lightning Indexer" sparse-attention SELECTION — the project's FIRST sparse candidate-selection primitive** (DeepSeek-V4-Flash W3). Two ops: (1) the weighted-MQA INDEXER LOGIT `logit[t,s] = Σ_h w[t,h]·ReLU(q[t,h]·k[s])` over the causal candidate window (the per-head **ReLU** is load-bearing — it is what makes the indexer a learned sparse SELECTOR, not a plain attention score), where `w[t,h] = weights_proj[t,h]·index_head_dim^-0.5·index_n_heads^-0.5`; and (2) the per-row **causal top-k** that keeps the `index_topk=512` highest-logit keys (short-context: every candidate, ascending; else top-k with -1 padding). Distinct from every dense/paged/MLA family, which score ALL keys — this one PICKS a sparse key subset the downstream MLA then attends over. W3 also lands the two 512-wide-MLA OUTPUT seams V2/V3 lack (per-head attention-**sink** softmax + **grouped output-LoRA** `wo_a` bmm→`wo_b`) as portable host references beside it | MQA logit `vllm/v1/attention/ops/triton_fp8_mqa_logits.py:120-156` (dot→×kv_scale→ReLU→×weights→Σheads); weight fold `vllm/model_executor/layers/sparse_attn_indexer.py:203-207`; top-k `sparse_attn_indexer.py:488-497` + short-context `vllm/models/deepseek_v4/attention.py:70-86,:813-831`; sinks `deepseek_v4/nvidia/flashinfer_sparse.py:777,:896`; grouped output-LoRA `deepseek_v4/nvidia/ops/o_proj.py:58-73` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_dsa.cpp](../src/vllm/model_executor/models/deepseek_v4_dsa.cpp) + [deepseek_v4_dsa.h](../include/vllm/model_executor/models/deepseek_v4_dsa.h): `DsaIndexerWeightFold` / `DsaIndexerLogits` / `DsaTopkSelect` / `SoftmaxWithSink` / `GroupedOutputLora` | **CPU UNIT GATE GREEN (2026-07-28, `-Wall -Werror -Wextra` 0-warn):** [test_deepseek_v4_dsa.cpp](../tests/vllm/models/test_deepseek_v4_dsa.cpp) **13/13 cases · 38 assertions** — hand-derived literal cases (the ReLU clip, the weight fold, short-context all-select, full top-k, tie→smaller-index, causal-window offset, sink probability mass, sink numerical stability, grouped-LoRA) + from-first-principles double-precision references on randomized shapes (indexer logits + grouped output-LoRA rel-L2 < 1e-6). Full-model gate is multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W3 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W3` | | `KERNEL-ATTN-DSA-COMPRESSOR` | **DeepSeek-V4 DSA COMPRESSOR + fp8_ds_mla KV-cache state — the second half of the sparse-attention stack** (DeepSeek-V4-Flash W4). Where `KERNEL-ATTN-DSA-SPARSE-INDEX` SELECTS keys, this POOLS + QUANTIZES them into the compressed latent the MLA reads and defines how it is cached across steps. Three ops: **(1)** the softmax-weighted window POOL — at a compress boundary the compressor gathers `(1+overlap)·compress_ratio` KV-state rows and computes, PER head-dim column, `softmax(score, dim=0)·kv` (each channel pools the window with its OWN weights — the load-bearing nuance), then RMSNorm; **(2)** the fused save-time APE add `score_state = score + ape[position % compress_ratio]`; **(3)** the **fp8_ds_mla** KV-cache STATE layout — the 512-wide latent split into a 448-wide NoPE part quantized to FP8 e4m3 with per-64 **UE8M0** power-of-two block scales (exponent `= ceil(log2(absmax/448))`, byte `= exp+127`) and a 64-wide RoPE part stored bf16, at a **576-byte** token stride with a padded **7+1** scale region — plus the dequant READ (`nope = e4m3·2^(byte-127)`, `rope = bf16`) | pool+RMSNorm `vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py:198-218`; save-time APE `common/ops/save_partial_states.py:92-101`; fp8_ds_mla store `fused_compress_quant_cache.py:220-297`; layout `deepseek_v4/compressor.py:307-309`; dequant READ cross-checked to SGLang `v0.5.15` `dsv4/dequant_k_cache.py:12-18,:122-136` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_compressor.cpp](../src/vllm/model_executor/models/deepseek_v4_compressor.cpp) + [deepseek_v4_compressor.h](../include/vllm/model_executor/models/deepseek_v4_compressor.h): `CompressorSaveScoreApe` / `CompressorPoolNorm` / `MakeFp8DsMlaLayout` / `Fp8DsMlaEncodeToken` / `Fp8DsMlaDecodeToken` | **CPU UNIT GATE GREEN (2026-07-29, Debug full-library build, 0-warn on the new TUs):** [test_deepseek_v4_compressor.cpp](../tests/vllm/models/test_deepseek_v4_compressor.cpp) **12/12 cases · 164 assertions** — hand-derived literal cases (APE modulo wrap; per-column softmax pool proven load-bearing via the column-ratio-survives-RMSNorm case; window masking; V4 layout 448/64/576/7+1; all-ones→UE8M0 byte 119 exact round-trip; value-3→byte 120; bf16 rope verbatim) + from-first-principles double-precision references (pool+norm rel-L2 < 1e-6; independent UE8M0 scale-byte recompute; encode→decode round-trip < 0.05 fp8 granularity). RED-first PROVEN: perturbing the scale bias `+127→+126` fails 4 cases / 135 assertions; revert restores 12/12. Honest gate form: hand-case + structural review vs vLLM+SGLang `file:line` (fixed-config 167B not constructible at a tiny shape ⇒ NOT a dumped-oracle rel-L2). Full-model gate multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + the fused device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W4 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W4` | | `KERNEL-MHC-SINKHORN` | **DeepSeek-V4 Manifold/Markov Hyper-Connections (MHC) — the Sinkhorn-normalized hc_mult-stream residual manifold** (DeepSeek-V4-Flash W5, the hardest V4 brick). V4 replaces the plain `residual + RMSNorm` stream with a `[tokens, hc_mult=4, hidden]` MANIFOLD of parallel residual streams, mixed at every attn/ffn boundary by a **doubly-stochastic** matrix and collapsed by a learned head. Four ops: **(1)** the **`hc_sinkhorn_iters=20` Sinkhorn** normalization of the hc_mult×hc_mult mixing matrix — a row-softmax seed (`+eps`), a col-norm, then `(iters-1)×[row-norm, col-norm]` toward a doubly-stochastic matrix (the AXIS ALTERNATION and the ITERATION COUNT are load-bearing at non-converged counts — RED-first proven); **(2)** the mHC **pre** mix — flatten the streams, project through `hc_*_fn` with a FOLDED weight-free RMSNorm `rsqrt(sqrsum/(hc·H)+rms_eps)`, split into pre/post/comb gates (`pre=σ+hc_eps`, `post=σ·hc_post_alpha(2.0)`, `comb=Sinkhorn`), collapse to the single `layer_input`, and optionally FOLD the model's attn_norm/ffn_norm RMSNorm; **(3)** the mHC **post** mix — fold the block output back into the manifold via the comb matrix (`Σ_i comb[i,j]·res[i,h]`) + the post gate; **(4)** the **hc_head** collapse — weight-free RMSNorm → `hc_head_fn` → sigmoid gate → weighted stream sum → one hidden vector. **EAGER-REF FINDING: corrects the W0 "ZERO eager reference upstream" premise** — the pinned vLLM DOES ship an eager PyTorch reference (`mhc/torch.py` `mhc_pre_torch`/`mhc_post_torch`, `triton.py` head collapse); four upstream impls (torch.py, tilelang_kernels.py `_sinkhorn_fwd`, tilelang.py, SGLang mhc.py) agree byte-for-byte on the Sinkhorn | mHC pre/post + Sinkhorn `vllm/model_executor/kernels/mhc/torch.py:56-106` (byte-identical `tilelang_kernels.py:126-153` `_sinkhorn_fwd`, `tilelang.py` `mhc_pre_big_fuse_with_norm`); head collapse `triton.py:108-140` + `tilelang.py:720-748`; constants `hc_post_alpha=2.0`/`hc_pre_eps=hc_sinkhorn_eps=hc_eps` `vllm/models/deepseek_v4/nvidia/model.py:818-821,:886-894,:1023-1041`; cross-checked SGLang `v0.5.15` `python/sglang/srt/layers/mhc.py:110-126` @ `555967922` | Portable host reference (device kernel + `DeepseekV4Model::Forward` assembly are W7 residuals) [deepseek_v4_mhc.cpp](../src/vllm/model_executor/models/deepseek_v4_mhc.cpp) + [deepseek_v4_mhc.h](../include/vllm/model_executor/models/deepseek_v4_mhc.h): `MhcSinkhorn` / `MhcPre` / `MhcPost` / `HcHeadCollapse` | **CPU UNIT GATE GREEN (2026-07-29, Debug full-library build, `-Wall -Werror -Wextra` 0-warn on the new TUs):** [test_deepseek_v4_mhc.cpp](../tests/vllm/models/test_deepseek_v4_mhc.cpp) **14/14 cases · 125 assertions** — hand-derived literal cases (all-zero Sinkhorn → uniform doubly-stochastic 1/hc; symmetric-2×2 fixed point `[[.75,.25],[.25,.75]]`; iteration-count load-bearing; MhcPre fn=0 gate midpoints; RMSNorm fold `[1,3]→[1,3]/√5`; MhcPost identity-comb + post-add; mix sums over the first comb index; hc_head fn=0 stream mean) + from-first-principles DOUBLE-PRECISION references (Sinkhorn/MhcPre/MhcPost/HcHead f32==f64 rel-L2 < 1e-5..1e-4; doubly-stochastic convergence to row/col sums=1). **RED-first PROVEN both levers:** perturb the Sinkhorn iteration count (`iters-1→iters-2`) fails 1 case/9 assertions AND swap a normalization axis fails 2 cases/12 assertions (caught by a dedicated SMALL-iteration-count gate, since at 20 iters the Sinkhorn has converged and ±1 is within tolerance); revert restores 14/14·125. Honest gate form: DERIVED-eager-reference + hand-case + structural review vs vLLM+SGLang `file:line` (fixed-config 167B not constructible at a tiny shape ⇒ NOT a dumped-oracle rel-L2). OPEN QUESTION: end-to-end bf16 residual/layer_input rounding between steps is a W7 device concern, not folded into these f32/f64 refs. Full-model gate multi-Spark-blocked (156.7 GiB); sqrtsoftplus/hash MoE (W6) + device kernel + forward assembly (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W5 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W5` | @@ -213,9 +214,16 @@ were removed while their specs and same-tool traces remain. Lifecycle stays ## Count invariants -- This table has exactly 35 practical kernel-family rows. +- This table has exactly 36 practical kernel-family rows. - Baseline lifecycle counts are 8 `ANCHOR-BACKFILL`, 0 `READY`, 4 `PARTIAL`, - 9 `ACTIVE`, 0 `GATING`, 1 `DONE`, and 13 `INVENTORIED`. + 10 `ACTIVE`, 0 `GATING`, 1 `DONE`, and 13 `INVENTORIED`. +- `KERNEL-DFLASH2-GROUPED-CONV` was added on 2026-08-19 (SPEC-DFLASH2 W2, + [#1314](https://github.com/mudler/vllm.cpp/issues/1314)) and is the tenth + `ACTIVE` row. It is a separate family from `KERNEL-DEPTHWISE-CONV1D` rather + than a variant of it: that kernel's weights are static per channel and its mask + is causal over the SEQUENCE, while this one's are projected per position from + the sublayer input, grouped, and masked over the QUERY BLOCK. It stays `ACTIVE` + rather than `DONE` because its CUDA arm has never compiled (spec `## Owed` O6). - The distinct `KERNEL-CPU-A76-Q8-DOT` compiler/assembly family (`GATING`, PR #79) was added on 2026-08-06; `scripts/check-agent-record.py` pins the total row count. diff --git a/.agents/specs/dflash2-spec-decode.md b/.agents/specs/dflash2-spec-decode.md index 3dbf2c25b..7a8e0f8ab 100644 --- a/.agents/specs/dflash2-spec-decode.md +++ b/.agents/specs/dflash2-spec-decode.md @@ -28,10 +28,18 @@ and its device path walk, the vocabulary top-k the selector consumes, and the reachable from `--speculative-config` through the loader and `ModelRegistry::Forward`, not merely constructible. +BOTH published DFlash2 checkpoints are in scope, because both resolve to the SAME +class: upstream registers one architecture, `DFlash2DraftModel -> qwen3_dflash2`, +and `z-lab/Qwen3.8-27B-DFlash2` and `z-lab/Muse-Glimmer-30B-DFlash2` both declare +`model_type` `qwen3`. What the second one adds is not a class but VALUES: +`block_size` 16 against the first's 8, and two of the three output scalars set +rather than defaulted ([#1327](https://github.com/mudler/vllm.cpp/issues/1327)). +An earlier revision of this section excluded "a second DFlash2 target family", +which was true about the CLASS and false about the work. + Out of scope: any change to the DFlash draft's own behaviour beyond the shared -`is_causal` rule; a second DFlash2 target family (upstream registers exactly one, -Qwen3); the DSpark lane; and any throughput claim, which `## Gates` defers with -its reason. +`is_causal` rule; the DSpark lane; and any throughput claim, which `## Gates` +defers with its reason. ## Upstream chain @@ -98,10 +106,28 @@ and checked against the header: `num_groups = 5120/16 = 320`, so the prepare/finish side rather than a tap. Three scalars upstream reads with a default — `input_embedding_scale`, -`output_multiplier`, `final_logit_softcapping` — are ABSENT from this config and -take 1.0, 1.0 and disabled. No published checkpoint exercises them, so the port -implements them and gates them synthetically rather than claiming checkpoint -coverage. +`output_multiplier`, `final_logit_softcapping` — are ABSENT from the Qwen3.8 +draft's config, which takes 1.0, 1.0 and disabled. + +**`z-lab/Muse-Glimmer-30B-DFlash2` sets two of them**, so they are +CHECKPOINT-EXERCISED rather than synthetic +([#1327](https://github.com/mudler/vllm.cpp/issues/1327)). Its `config.json` +(sha256 `cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9`, read +2026-08-19) declares `output_multiplier 0.19611613513818404` and +`final_logit_softcapping 20.0`, together with `block_size` 16, hidden 6656 (so +`num_groups` 416 and a 1664-wide `kernel_projection`), `rope_theta` 500000.0 +nested under `rope_parameters`, and vocab 202048. Both scalars are applied to the +candidate VALUES in `compute_candidates` BEFORE the selector scores them +(`qwen3_dflash2.py` `DFlash2Qwen3ForCausalLM.compute_candidates` @ the PR head), +so a wrong value reorders the top-K and moves acceptance without raising — the +`is_causal` failure mode one layer up. This is the same pair Muse Glimmer's text +tower already needed here: `docs/USAGE.md` records that the released 30B config +carries both while the GGUF and the DFlash drafter each omit some, and that both +used to run a quietly different model. `input_embedding_scale` remains +unexercised by both published drafts and stays synthetic. + +An earlier revision of this section said no published checkpoint exercised any of +the three. That was measured on the Qwen3.8 draft alone. ## Our baseline @@ -138,7 +164,7 @@ Absent, and owed by this row: (`include/vllm/config/speculative.h:115-138`) with its loader refusal (the classification helper `src/vllm/entrypoints/model_loader.cpp::ReadDflashDraftArchitectures` and - `src/vllm/entrypoints/model_loader.cpp::RefuseDflash2Draft`, cited by SYMBOL + `src/vllm/entrypoints/model_loader.cpp::CheckDflash2DraftArm` (named `RefuseDflash2Draft` until W2 split the two container arms), cited by SYMBOL because the line range this spec first carried was stale two merges later and `scripts/check-symbol-anchors.py` states it cannot verify a line citation). The identical gap for @@ -157,7 +183,7 @@ Absent, and owed by this row: | Upstream | Ours | Note | |---|---|---| -| `qwen3_dflash2.py` conv | `vt::DFlashGroupedConv` op + `src/vllm/model_executor/models/qwen3_dflash2.cpp` | CPU reference first, CUDA after, as `KERNEL-ATTN-DFLASH-BLOCK` did | +| `qwen3_dflash2.py` conv | `vt::DFlashGroupedConv` op, kernel-matrix row `KERNEL-DFLASH2-GROUPED-CONV`, wrapped into the existing `src/vllm/model_executor/models/qwen3_dflash.cpp` layer bodies | CPU reference first, CUDA after, as `KERNEL-ATTN-DFLASH-BLOCK` did. LANDED in W2. The conv wraps sublayers of the SHIPPED DFlash block rather than a new file, because upstream subclasses `DFlashQwen3DecoderLayer` and overrides only `forward`; a parallel `qwen3_dflash2.cpp` copy of that body is what AGENTS.md `## Shared seams` forbids | | `qwen3_dflash2.py` selector | same file, plus a `vt` lattice op | `_score_edges` is one einsum; it is not the cost | | `_topk` / FlashInfer radix | extend `src/vt/cuda/cuda_sample.cu:297-506` to emit pairs | D2 | | `dflash2/speculator.py` walk kernel | `src/vllm/v1/worker/gpu/spec_decode/dflash2/speculator.cpp` + a CUDA walk kernel | D3: device from day one | @@ -202,7 +228,15 @@ Ours, red-first, beyond the ports: **Correctness, before any speed number.** - G1: conv and lattice against upstream's references, CPU then CUDA, at the - checkpoint's real shapes (taps 2, group 16, block 8, K 16, rank 256). + checkpoints' real shapes (taps 2, group 16, K 16, rank 256). + **G1 runs at BOTH published block shapes, 8 and 16.** Upstream's own reference + test parametrises `block_size` 5 and 8 to cover the power-of-two branch of the + position mask (`pos & (block-1)` against `pos % block`); 16 is the shape a real + checkpoint ships (`z-lab/Muse-Glimmer-30B-DFlash2`) and neither upstream + parameter reaches it ([#1327](https://github.com/mudler/vllm.cpp/issues/1327)). + W2 discharges the conv half of G1 on CPU at 5, 8 and 16; the CUDA half is + written and registered but UNVERIFIED (no `nvcc` on the authoring host) and is + owed to a GPU lease. - G2: draft-token identity against vLLM at PR head `19c93519` on `z-lab/Qwen3.8-27B-DFlash2` over `Qwen/Qwen3.8-27B`, identical prompts and identical k, greedy. The DFlash near-tie envelope applies: `SPEC-DFLASH` @@ -321,6 +355,31 @@ reviewer who mutates the guarantee rather than reading it. no `is_causal` at all, and `z-lab/Qwen3.8-27B-DFlash2` declares it as a JSON boolean. Reconcile onto upstream's coercion if a checkpoint ever spells it as a string, and change the GGUF arm in the same edit or not at all. +- **D9 — the row gates the OUTPUT SCALARS against a checkpoint that SETS them, + not against defaults.** Discovered after this spec landed + ([#1327](https://github.com/mudler/vllm.cpp/issues/1327)); the brief that + proposed it called it D8, which was already taken by the `is_causal` coercion + divergence above, so it is D9 here. A port that reads all three with + `.get(key, default)` passes every gate built from the Qwen3.8 draft alone, + because that draft sets none of them — such a gate measures the default path + and reports it as coverage. W3 owns the scalars and must gate them on + `z-lab/Muse-Glimmer-30B-DFlash2`'s values. +- **D10 — the DFlash2 refusal MOVES from startup to the first propose, on the + safetensors arm only.** W2 decision, 2026-08-19. W1 refused a + `DFlash2DraftModel` draft before any weight was read, when both mechanisms were + missing. W2 implements one of them, and keeping the startup refusal would leave + every line of it unreachable from any production entry point — AGENTS.md + `## Nothing lands dead` — with the conv gated only by tests that construct it, + which is `.agents/reachability.md`'s test-only driver. So a safetensors DFlash2 + draft is now ADMITTED: it loads its conv weights, runs the conv in all three + layer bodies, and is refused BY NAME at `RefuseDflash2CandidateSelector`, after + the block forward and before anything samples. Cost: the refusal arrives at the + first generated token rather than at startup. It is paid down by a STARTUP + NOTICE from `CheckDflash2DraftArm` naming the mechanism that runs, the one that + does not, the wave that owns it and the issue, so nothing is a surprise. The + GGUF arm KEEPS the startup refusal, because its drafter arm (W5) has no conv + weight path at all and admitting the file would load a DFlash1 draft out of a + DFlash2 checkpoint. ## Owed @@ -331,181 +390,162 @@ slice to land unreached only when this list, the commit body and the pull request body all name it, so this section is the record that permission depends on and not a summary. -- **O1 — the `is_causal` half of W1 is INERT at its own merge commit, on both - container arms.** Owner: this row, discharged by W2. Issue +**W2 DISCHARGED O1, O2, O3 and O4.** They are kept below, struck through in +prose rather than deleted, because the reason each existed is what a later reader +needs and `.agents/completed/` is for superseded documents rather than for four +list items. + +- **O1 — DISCHARGED by W2.** The `is_causal` half of W1 was INERT at its own + merge commit, on both container arms: every artifact that declared the key also + declared the DFlash2 markers W1 refused, so no checkpoint W1 admitted could + take any of the three arms. W2 admits `z-lab/Qwen3.8-27B-DFlash2` — which + declares `is_causal false` beside five `sliding_attention` layers — through + `CheckDflash2DraftArm`, and `MakeQwen3DFlashDraftConfig` can now parse it (O3), + so the rule is REACHED by a published checkpoint. Gated in + `tests/vllm/models/test_qwen3_dflash2_draft.cpp`, which drives the whole + published `config.json` through the production builder and asserts all five + layers resolve non-causal. The GGUF arm's inertness is unchanged and moves with + W5. +- **O2 — DISCHARGED IN PART by W2, and the remainder is O5.** W1's fresh reviewer + proved the loader's own call sites were not gated for the causality carry. + `LoadQwen3DFlash` is now driven from a test over a REAL on-disk safetensors + shard with the published tensor names, which is the function `LoadDflashDraft` + calls, so the weight half is gated. What is still not gated is the part of + `LoadDflashDraft` that lives inside the loader's anonymous namespace — see O5. +- **O3 — DISCHARGED by W2.** `MakeQwen3DFlashDraftConfig` could not parse either + published DFlash2 `config.json`: it did `c.at("rope_theta")` and + `c.at("block_size")` while both drafts nest them as `rope_parameters.rope_theta` + and `dflash_config.block_size`. Both are now fallbacks rather than + replacements — the flat spelling is read FIRST, so every DFlash1 draft is + unchanged — and the RoPE default is upstream's own + `set_default_rope_theta(config, default_theta=1000000)`. Red-first: the case + driving the published Muse-Glimmer document threw + `[json.exception.out_of_range.403] key 'rope_theta' not found`, quoted in the + W2 commit body. +- **O4 — DISCHARGED by W2, as ONE change rather than three.** (a) `layer_types` + is optional, mirroring `getattr(config, "layer_types", None)`, so + `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` no longer throws + `key 'layer_types' not found` and #1366's `use_swa` rule resolves for it — + five layers, every one `causal=0` with `sliding_window=1024`, which is + upstream's docstring row. (b) `dflash_config.attention_sink_bias` (and the + top-level `add_swa_attention_sink_bias` upstream falls back to) is REFUSED BY + NAME, because this lane has no attention sink and landing (a) alone would have + converted a loud parse error into a quiet wrong answer. A FALSY value is + upstream's default and is not refused. (c) O4's third fact stands unchanged and + is not repaired by this wave: MiMo's target `MiMoV2ForCausalLM` is still + `INVENTORIED` and unimplemented, so no production entry point can serve the + model that draft heads. The parse is correct and the refusal is loud; the + drafter is still not runnable, and that belongs to the MiMo model row. + +- **O5 — `LoadDflashDraft`'s own DFlash2 lines are not gated.** Owner: this row, + discharged by W4. Issue + [#1314](https://github.com/mudler/vllm.cpp/issues/1314). Mutation-proven by W2 + on 2026-08-19: deleting `draft->weights.conv_block_size = draft->k + 1;` from + `LoadDflashDraft` (`src/vllm/entrypoints/model_loader.cpp`) COMPILES CLEAN and + leaves both focused suites GREEN (`test_qwen3_dflash2_draft` 16/16, 108 + assertions; `test_dflash2_draft_routing` 12/12, 33 assertions). What holds the + conv's block today is the test setting the field itself, so the gate measures + `LoadQwen3DFlash` and the forward, not the loader that wires them. It is not + repaired where it was found for the reason W1's O2 gave and W2 confirmed: + `LoadDflashDraft` is `static` inside the loader's anonymous namespace, and an + entry-point gate on it has to load a draft off a LIVE target, sharing that + target's `embed_tokens` and `lm_head`. W4 brings the speculator wiring that + makes such a gate constructible. The consequence if it regressed is bounded and + named: the conv would mask its taps against the checkpoint's DEFAULT block + instead of the resolved `k`, which is invisible unless the two differ — and + acceptance-only, token-invisible, when they do. +- **O6 — the CUDA arm of `vt::DFlashGroupedConv` is UNVERIFIED.** Owner: this + row, discharged by the operator's GPU lease before W6. Issue + [#1314](https://github.com/mudler/vllm.cpp/issues/1314). The kernel and its + registration are written and reviewed + (`src/vt/cuda/cuda_ops.cu`, `DFlashGroupedConvKernel` / + `DFlashGroupedConvKernelCuda`), and the CUDA==CPU bit-identity case exists and + is written to run + (`tests/vt/test_ops_dflash2_grouped_conv.cpp`, six shapes covering both + published blocks in bf16 and the modulo arm in f32). It has NEVER COMPILED: + the authoring host has no `nvcc`, so the CUDA case reports + `no CUDA backend; skipping CUDA dflash2-grouped-conv parity` and the file's + 9410 assertions are all CPU. Two specific things are unproven rather than + merely unrun: that the kernel compiles at all, and that + `__fadd_rn`/`__fmul_rn` plus `ResRound` reproduce the CPU reference BIT-FOR-BIT + on the f32 arm, where the intrinsics are the only thing forbidding an FMA + contraction the CPU build pins off. This is named here rather than reported as + a pass. +- **O7 — the runner's own selector-refusal call site is not gated.** Owner: this + row, discharged by W4. Issue [#1314](https://github.com/mudler/vllm.cpp/issues/1314). - `ResolveQwen3DFlashAttnModes`' top-level arm, `MakeQwen3DFlashDraftConfig`'s - carry of the key, and `MakeDflashGgufConfig`'s read of - `dflash.attention.causal` are all live code with unit gates, and no checkpoint - this commit ADMITS can take any of them. The reason is the other half of the - same wave: every artifact that declares the key also declares the DFlash2 - markers W1 refuses — `z-lab/Qwen3.8-27B-DFlash2` declares `is_causal false` - beside `architectures: ["DFlash2DraftModel"]`, and - `z-lab/Qwen3.8-27B-DFlash2-GGUF` declares `dflash.attention.causal` beside - `dflash.selector_rank` — and `RefuseDflash2Draft` throws on both before any - config is built. Every published DFlash1 artifact declares neither key, so it - takes the legacy arm exactly as before, which is the inertness the wave's - gates assert on purpose. This is `.agents/reachability.md`'s "unselected - branch" shape: the branch is reached by construction in a test and by no input - the production entry point accepts. It becomes live in W2, which is the wave - that lifts the refusal for the parts it implements. W1 lands it anyway because - splitting a refusal from the rule that makes the refused checkpoint correct - would land the refusal alone and leave the rule to be rediscovered. -- **O2 — the loader's production call sites are not gated for the causality - carry.** Owner: this row, discharged by W2. Issue - [#1314](https://github.com/mudler/vllm.cpp/issues/1314). Mutation-proven by - W1's fresh reviewer: appending `draft->config.raw.erase("is_causal");` after - the `MakeQwen3DFlashDraftConfig` call in `LoadDflashDraft` - (`src/vllm/entrypoints/model_loader.cpp`, safetensors arm) and after the - `MakeDflashGgufConfig` call (GGUF arm) each COMPILES CLEAN and leaves both - focused suites GREEN. What holds the carry is the direct unit call on the two - builders, so the gate measures the builders and not the loader that uses them. - It is not repaired where it was found because an entry-point gate on this path - has to load a draft, and a draft load needs real draft weights: `LoadDflashDraft` - reads the config and the shards in one function and shares `embed_tokens` and - `lm_head` off a live target. W2 brings the DFlash2 fixture weights that make - such a gate constructible; before then the only reachable form is another unit - call, which is the thing that already fails to hold. -- **O3 — `MakeQwen3DFlashDraftConfig` cannot parse the published DFlash2 - `config.json` at all.** Owner: this row, discharged by W2. Issue - [#1314](https://github.com/mudler/vllm.cpp/issues/1314). Found by W1's fresh - reviewer and confirmed against the file on 2026-08-19: the builder does - `c.at("rope_theta")` and `c.at("block_size")`, and `z-lab/Qwen3.8-27B-DFlash2` - nests them as `rope_parameters.rope_theta` and `dflash_config.block_size`, with - neither key present at the top level. Both `at` calls throw, so the builder - cannot construct a config for the checkpoint this row exists to run. It is - invisible today because W1 REFUSES that checkpoint earlier, at - `RefuseDflash2Draft`, which runs before any config is built. It is a W2 - blocker rather than a W1 defect: the builder is only ever asked to parse a - DFlash2 config once W2 lifts the refusal, and repairing it in W1 would land a - parse arm for a shape nothing feeds. `transformers` moved RoPE settings under - `rope_parameters`, so the fix is a fallback and not a replacement — DFlash1 - checkpoints still carry the flat spelling. -- **O4 — the `use_swa` causality repair (#1366) is UNREACHED, and the config - builder cannot parse a draft that declares no `layer_types`.** Owner: this row, - discharged by W2. Issue - [#1314](https://github.com/mudler/vllm.cpp/issues/1314). Found by W1's second - fresh review and repaired here as a RECORD correction rather than as code, - because making it reachable is not a small and clear change. Three facts, each - checked on 2026-08-19: - (a) `MakeQwen3DFlashDraftConfig` does `c.at("layer_types")`, and upstream reads - `getattr(config, "layer_types", None)` (`qwen3_dflash.py:134`, and `:66` in - `_dflash_layer_causal`, @ - vllm-project/vllm#52816 head `19c9351904df4c63042671bc67a866ca48dc7d6f`), so an - absent key is upstream's `None` and is this builder's raw - `[json.exception.out_of_range.403]`. Mirroring that one `getattr` is a - three-line change, and `layer_types` is the ONLY key of MiMo's real - `dflash/config.json` this builder is missing -- every other name it reads is - present at the top level there, which is what separates this entry from O3. - Both halves of that are MEASURED rather than argued, on the published file - (sha256 `2ed5a998f5f57e00a9fe14d2b3e767f06e49462a97eb09d80c927e112a585c9e`) - driven through the production builder by a scratch program on 2026-08-19. As - shipped it prints `THREW: [json.exception.out_of_range.403] key 'layer_types' - not found`. With the three-line fallback applied in a scratch copy, restored - byte-for-byte afterwards and verified by sha256, the same file builds and - `ResolveQwen3DFlashAttnModes` answers five layers, every one `causal=0` with - `sliding_window=1024` -- exactly upstream's `layer_types=None` + `use_swa=True` - docstring row, and the opposite of what this engine answered before #1366. So - the rule and the fallback are both right; what is missing is anything that can - feed them. - (b) It would still buy no reachability. `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` - is the only published draft with `use_swa` and no `layer_types`, and its target - `MiMoV2ForCausalLM` is `INVENTORIED`, unassigned and unimplemented in - `.agents/model-matrix.md`, so no production entry point can serve the model - that draft heads. A drafter is not reachable without its target. - (c) It would make a WRONG path selectable where a loud parse failure - stands today. - Upstream reads `dflash_config.attention_sink_bias` and passes a per-head sink - bias into its `Attention` (`qwen3_dflash.py:309-313` and `:240-257` @ that - head); MiMo's config sets it true and this lane has no attention sink at all, - so a MiMo draft that parsed would load with the sinks silently absent -- - acceptance-only and token-invisible, the exact class this row exists to remove. - Landing (a) alone converts a loud parse error into a quiet wrong answer. - W2 discharges this with O3, and it must land the `layer_types` fallback, a - named refusal for `dflash_config.attention_sink_bias`, and the loader-entry - gate O2 owes, as one change rather than three. + `RefuseDflash2CandidateSelector` has TWO production call sites: + `DflashProposeBlock` (gated — deleting the call turns + `test_qwen3_dflash2_draft` red, 1 case / 1 assertion) and + `GPUModelRunner::propose_drafts_block` (NOT gated: entering it needs a + constructed `GPUModelRunner` with a loaded target, a KV cache and a spec + config). The two call sites are one line apart in intent and are easy to keep + in step, and the ungated one is the one a user actually arrives through. W4 + wires the DFlash2 speculator and is the wave that can enter that path. ## Now -`SPEC-DFLASH2` is `ACTIVE`. W1 landed on 2026-08-19: the `DFlash2DraftModel` -route and D4's `is_causal` precedence, both red-first and both mutation-proven on -CPU. No DFlash2 mechanism landed with it, and none is claimed. - -What W1 ships is a REFUSAL. A draft whose `config.json` declares -`DFlash2DraftModel` is refused at startup, before any weight is read, with both -missing mechanisms named — from the dflash branch of -`LoadedEngine::ResolveSpecConfig` and again at the top of -`LoadedEngine::FromModelDir`, which is the site that matters, because the dflash -draft load runs there BEFORE the constructor's resolution. Refusing rather than -loading is the whole point: a DFlash2 checkpoint carries DFlash1's entire tensor -set, so the DFlash1 lane would load it with nothing missing and draft worse -tokens with no visible symptom. - -D4 landed with it, in both halves, and is NOT YET REACHED. `ResolveQwen3DFlashAttnModes` -resolves a top-level `is_causal` ahead of `dflash_config.causal` and ahead of the -legacy `layer_types` rule, and `MakeQwen3DFlashDraftConfig` — moved out of the -loader's anonymous namespace so the key it carries is gateable at all — copies the -key off the draft's own `config.json`. A resolution that reads a key the config -builder drops is half a port, and only the two together make the rule reachable. -Reachable in principle: no checkpoint W1 ADMITS declares the key, because every -artifact that declares it also declares the DFlash2 markers the same commit -refuses. `## Owed` O1 records that precisely, O2 records that the loader's own -call sites are not gated for the carry, and W2 discharges both. What W1 asserts -about D4 is therefore a unit guarantee plus the inertness of the DFlash1 lane, -and no more than that. - -W1 also covers the GGUF drafter, which the spec's own W1 text did not name and -which the `config.json`-keyed classification cannot see. `z-lab/Qwen3.8-27B-DFlash2-GGUF` -@ `57ab3265056d4024870b0621cfc2c127537020ed` writes `general.architecture = "dflash"`, -byte-identical to a DFlash1 drafter, and a GGUF carries no `architectures` array -at all — so `qwen3_dflash_gguf.cpp` would have loaded it as DFlash1 with no -error. The discriminator is therefore the DFlash2-only metadata -(`dflash.selector_rank`, `dflash.selector_top_k`, `dflash.conv_kernel_size`), and -`dflash.attention.causal` is the GGUF spelling of `is_causal` and resolves in the -same precedence. Both were read off the published file on 2026-08-19; the shipped -DFlash1 drafter `muse-glimmer-30b-gguf/dflash-kquant.gguf` carries none of those -keys and is unchanged. This is a REFUSAL on the GGUF axis, not the GGUF drafter -ARM, which stays W5. - -W1's fresh review also raised [#1366](https://github.com/mudler/vllm.cpp/issues/1366) -against the same function, and it is FIXED IN FLOW rather than deferred, per -AGENTS.md. It is pre-existing and it is D4's failure class one arm over. The -legacy fallback read the RESOLVED `is_sliding`, which `dflash_config.use_swa` -forces true on every layer; upstream reads the DECLARED `layer_types` -(`bool(layer_types) and layer_types[i] == "sliding_attention"`, -`qwen3_dflash.py:66-67` @ the PR head), and states the consequence as a row of -its own `_resolve_layer_attention` docstring table -- `layer_types=None` + -`use_swa=True` -> causal False -- naming `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` -as the published checkpoint of that shape. Such a DFlash1 draft therefore ran -every layer CAUSAL here and non-causal upstream, with the verify lossless and -only acceptance moving. It went uncaught because upstream's parametrize table -has no `use_swa` row either, so the ported cases were faithful to the ported -table and silent about the arm. The same issue's second half is the coercion: -`is_causal` was honoured only as a JSON boolean, while upstream tests presence -and coerces, and the GGUF arm's `KvI64` already took every integer width -- so -`"is_causal": 0` fell through in silence and the two containers disagreed with -each other. Both halves are repaired red-first and mutation-proven, and -NEITHER IS REACHED. An earlier revision of this section, and `4941dfbfe`'s commit -body, claimed the `use_swa` half was -- "REACHED today, because the checkpoints it -governs are DFlash1 ones this engine already admits". That was wrong on two -independent counts, both established by W1's second fresh review on 2026-08-19 and -confirmed against the published files and against this tree. -`XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` is the only published draft of the governed -shape, and it declares NO `layer_types`, while `MakeQwen3DFlashDraftConfig` does -`c.at("layer_types")` and throws `[json.exception.out_of_range.403] key -'layer_types' not found` before any causality is resolved. Its target -architecture `MiMoV2ForCausalLM` is `INVENTORIED` and unassigned in -`.agents/model-matrix.md`, so this engine cannot serve the model that draft -heads, whatever the builder does. The GGUF arm cannot reach the rule either: -`MakeDflashGgufConfig` never writes `use_swa` and always fills `layer_types` from -the sliding-window pattern. The three other published DFlash1 drafts -(`z-lab/Qwen3.6-27B-DFlash`, `z-lab/Qwen3.5-9B-DFlash`, -`z-lab/gemma-4-31B-it-DFlash`) all declare `layer_types` and no `use_swa`, so the -repair leaves their resolution byte-for-byte unchanged -- the inertness half, -which is the only thing W1 asserts about either half. `## Owed` O4 records the -gap, names W2 as its owner, and states why the reachability repair was not -attempted in this flow. - -Next action: W2, the grouped dynamic convolution, CPU reference first, against -the checkpoint's real shapes (taps 2, group 16, block 8). W2 must also discharge -`## Owed` O1, O2, O3 and O4, and O3 and O4 are blockers rather than cleanups. +`SPEC-DFLASH2` is `ACTIVE`. W1 landed on 2026-08-19 (the route and D4's +`is_causal` precedence). **W2 landed on 2026-08-19: the grouped dynamic depthwise +convolution, REACHED.** + +**What W2 ships is a mechanism, where W1 shipped a refusal.** `vt::DFlashGroupedConv` +is the project's first grouped dynamic depthwise convolution: +`out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, with tap +`t` contributing only where `(i mod block) >= t`, `g(c) = c / group_size`, and +`base_kernel` dim 0 the prepare/finish SIDE rather than a tap. The CPU kernel is +the authoritative reference and rounds to the tensor dtype after each step, as +upstream's bf16 chain materializes it — which is what lets the CUDA mirror be +asserted BIT-IDENTICAL rather than within an envelope. Both of upstream's +position-mask arms are ported (`pos & (block-1)` and `pos % block`) and gated at +block 5, 8 and 16. + +**The refusal MOVED so that the conv could be reached.** A safetensors +`DFlash2DraftModel` draft is now admitted at `CheckDflash2DraftArm`, loads its +`attention_conv`/`mlp_conv` tensors through `LoadQwen3DFlash`, runs the conv in +ALL THREE of the draft's layer bodies — `ForwardBlockLogits`, +`ForwardWithCtxKVDev` and `ForwardPagedBody`, the last being what the production +decode path reaches through `ForwardBlockLogitsWithDeviceKV` — and is then +refused BY NAME at `RefuseDflash2CandidateSelector`, after the block forward and +before anything samples. `## Risks/decisions` D10 carries the decision and its +cost; a startup NOTICE names the boundary so the later refusal is not a surprise. +The GGUF arm keeps its startup refusal and moves with W5. + +**Reachability was measured, not argued, and it cost two gate repairs.** The +first version of the model-level gate could not tell one missing call site from +none: it activated both convs at once, so deleting only the context-aware body's +`attention_conv` left the suite GREEN. The second could not see the SIDE: forcing +`args.side` to 0 in the kernel left the model suite GREEN. Both were found by +running the mutation rather than by reading the test, and both were repaired +before the wave landed — each conv is now driven ALONE through each body, and the +two sides are separated by `base_kernel[side]` scalars against a common identity +baseline. The final mutation set turns the focused suites red for: each body's +call sites (three separate mutations), the side index, the block mask, the group +map, the `rope_parameters` fallback, the `dflash_config.block_size` fallback, the +`layer_types` fallback, the `attention_sink_bias` refusal, the uniform-block +guard, the `DflashProposeBlock` refusal call, and restoring W1's startup refusal. + +**Four `## Owed` entries are discharged and three are new.** O1 (the `is_causal` +rule was inert), O2's weight half, O3 (`MakeQwen3DFlashDraftConfig` could not +parse either published DFlash2 config) and O4 (`layer_types`, plus the +`attention_sink_bias` refusal that had to land with it) are closed. O5 records +that `LoadDflashDraft`'s own `conv_block_size = k + 1` is UNGATED and +mutation-proven so; O6 records that the CUDA arm has never compiled on this host +and is owed to a GPU lease; O7 records that the runner's selector-refusal call +site is not gated. None of the three is a claim wearing a pass. + +**#1327 is corrected in this wave.** `## Upstream chain` said no published +checkpoint exercised `input_embedding_scale`, `output_multiplier` or +`final_logit_softcapping`. `z-lab/Muse-Glimmer-30B-DFlash2` sets +`output_multiplier 0.19611613513818404` and `final_logit_softcapping 20.0`, and +ships `block_size` 16 against the 27B's 8. Both scalars are applied to candidate +VALUES before the selector scores them, so a wrong one reorders the top-K and +moves acceptance without raising. `## Scope`'s exclusion of "a second DFlash2 +target family" is dropped (upstream registers ONE class and both checkpoints +declare `model_type` `qwen3`), `## Gates` G1 now requires both block shapes, and +`## Risks/decisions` D9 records that the scalars must be gated against the +checkpoint that sets them rather than against defaults. + +Next action: W3, the candidate selector — the lattice op, the codebooks in the +loader, and the top-k that EMITS pairs (D2). It is the wave that lifts the +refusal W2 leaves behind, and D9 binds it to Muse Glimmer's scalars. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index bf7257e89..7b72a6d2c 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -425,7 +425,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE 35B-A3B: **0.835x** paired on kairos-17dd (matched 89 tokens, warm oracle cache). Prior 0.957-0.989 came from a different machine with a cold oracle (#442) | `ACTIVE` | | DSpark block floor | Qwen3.8-27B + `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` | a `k` below the draft's block is refused instead of drafted; the run gate that exhibits the garbling is **owed** and needs a GPU lease (#1225) | `ACTIVE` | | DSpark draft routing | Qwen3.8-27B + `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` | **PENDING**, no number. The token-exact run gate needs the 2.53 GiB draft and GPU time, and neither authority is recorded; only the CPU classification gate has run (`.agents/specs/dspark-qwen3-routing.md` §6) | `ACTIVE` | -| DFlash2 route and causality | `Qwen/Qwen3.8-27B` + `z-lab/Qwen3.8-27B-DFlash2` | **PENDING, no number admissible.** W1 lands a refusal and no mechanism, so nothing is timeable; acceptance (G3, SAME-TRAJECTORY) reads before any ratio | `ACTIVE` | +| DFlash2 route and causality | `Qwen/Qwen3.8-27B` + `z-lab/Qwen3.8-27B-DFlash2` | **PENDING, no number admissible.** W2 landed the grouped convolution; the draft is still refused at the candidate selector, so no step exists to time. Acceptance (G3, SAME-TRAJECTORY) reads first | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index ef8d10d83..bebf681e7 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -385,7 +385,7 @@ a spike while its user-facing serving surface is finalized. **DSpark draft routing** (`SPEC-DSPARK-QWEN3-ROUTING`, ACTIVE) makes the loader classify a DSpark draft from the draft's own `config.json` before it resolves anything else. `Qwen3DSparkModel`, `Gemma4DSparkModel` and — ahead of the pin, mirroring vllm#52197 — `DSparkDraftModel` with `model_type` `qwen3` take the landed Qwen3 lane; a draft that resolves to the DeepSeek-V4 DSpark lane is refused BY NAME instead of being rewritten into a stub. CPU-gated only: the token-exact run gate against the pinned oracle waits on a draft download and GPU time that are not authorized, so it stays owed (#1193). -**DFlash2** (`SPEC-DFLASH2`, ACTIVE, [#1314](https://github.com/mudler/vllm.cpp/issues/1314)) has landed its first wave and none of its mechanism. Upstream carries DFlash2 as a second architecture beside DFlash rather than as a change to it, so a `DFlashDraftModel` checkpoint keeps resolving exactly as it does today; what the new `DFlash2DraftModel` adds is a grouped dynamic depthwise convolution inside each draft block and a candidate selector that replaces the per-slot argmax with a scored path walk over the target head's top-K. Neither is implemented, so a draft that declares `DFlash2DraftModel` is now REFUSED at startup, before any weight is read, with both missing parts named. It is refused rather than loaded because a DFlash2 checkpoint carries DFlash1's whole tensor set: the DFlash1 lane would take it with nothing missing and draft worse tokens with no visible symptom, since the verify is lossless and only acceptance falls. The refusal covers the GGUF drafter too, which is the case the architecture string cannot reach: the published DFlash2 GGUF declares the same `dflash` architecture a DFlash1 drafter does, so it is identified by the convolution and selector metadata only it carries. The second half of the wave is the causality rule — a top-level `is_causal` now decides every layer ahead of `dflash_config.causal` and ahead of the `layer_types` default, which is what the published checkpoint (all five layers `sliding_attention`, `is_causal false`) depends on; no DFlash1 checkpoint declares the key, in either container, so their resolution is unchanged. The port is BEYOND-PIN on an OPEN upstream pull request and does not move the parity pin. No speed number is claimed, and none is admissible before the acceptance gate reads. +**DFlash2** (`SPEC-DFLASH2`, ACTIVE, [#1314](https://github.com/mudler/vllm.cpp/issues/1314)) has landed its route, its causality rule and the FIRST of its two mechanisms — the grouped dynamic depthwise convolution — and none of the second. A safetensors `DFlash2DraftModel` draft now loads, runs that convolution around every attention and every MLP sublayer of every draft layer, and is refused BY NAME when the candidate selector would have to choose, with a notice at startup saying so in advance. A GGUF DFlash2 drafter is still refused at startup, because its weight path does not exist yet. The paragraph below describes the wave that shipped the refusal and remains accurate about everything except where the refusal now lands. Upstream carries DFlash2 as a second architecture beside DFlash rather than as a change to it, so a `DFlashDraftModel` checkpoint keeps resolving exactly as it does today; what the new `DFlash2DraftModel` adds is a grouped dynamic depthwise convolution inside each draft block and a candidate selector that replaces the per-slot argmax with a scored path walk over the target head's top-K. Neither is implemented, so a draft that declares `DFlash2DraftModel` is now REFUSED at startup, before any weight is read, with both missing parts named. It is refused rather than loaded because a DFlash2 checkpoint carries DFlash1's whole tensor set: the DFlash1 lane would take it with nothing missing and draft worse tokens with no visible symptom, since the verify is lossless and only acceptance falls. The refusal covers the GGUF drafter too, which is the case the architecture string cannot reach: the published DFlash2 GGUF declares the same `dflash` architecture a DFlash1 drafter does, so it is identified by the convolution and selector metadata only it carries. The second half of the wave is the causality rule — a top-level `is_causal` now decides every layer ahead of `dflash_config.causal` and ahead of the `layer_types` default, which is what the published checkpoint (all five layers `sliding_attention`, `is_causal false`) depends on; no DFlash1 checkpoint declares the key, in either container, so their resolution is unchanged. The port is BEYOND-PIN on an OPEN upstream pull request and does not move the parity pin. No speed number is claimed, and none is admissible before the acceptance gate reads. **DeepSeek-V4 native MTP** (`DeepSeekV4MTPModel`, ACTIVE — W1 self-spec wiring, 2026-07-30) has its nextn draft head wired to the same lossless spec-decode path. diff --git a/docs/USAGE.md b/docs/USAGE.md index a50e50e77..0a5098fa1 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2688,7 +2688,7 @@ a stop token early. | `--reasoning-parser ` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `` ends reasoning with no ``. `auto` never selects it — a generic `` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all | | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | `--offload-config ''` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend, an unknown TOP-LEVEL key (the four legal ones are `offload_backend`, `uva`, `prefetch` and `vllm_cpp`) or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Enabling it fails startup on every model today**: no loader consults the offloader, so the engine refuses the configuration by architecture name rather than accept a budget that frees nothing. A config that leaves offloading disabled still parses and reports normally. On unified memory such as GB10 offload cannot help at all, because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md). The same document also carries the **`vllm_cpp` key**, which governs the tier BELOW this one — weights borrowed out of the file mapping rather than moved to host RAM — and which is live rather than refused: see [Streaming routed experts from disk](#streaming-routed-experts-from-disk-capacity-mode). A `vllm_cpp`-only document does not enable vLLM's offload backends and is not subject to the refusal above. The flag is accepted by `vllm-server` (the generate/chat and the pooling/embedding paths), by `vllm-cli`, and by the C ABI; the server's transcription-only path REFUSES it by name, because that path builds no engine and could only accept the document and ignore it ([#1195](https://github.com/mudler/vllm.cpp/issues/1195)) | -| `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. For `mtp`, `num_speculative_tokens` sets the draft DEPTH and defaults to the checkpoint's `mtp_num_hidden_layers`, which is 1 on both gate checkpoints, so the default is unchanged. A value above it must be a multiple of it, mirroring vLLM. Depth cannot move the emitted tokens under greedy decoding, and no speed number is claimed above k=1 yet ([#81](https://github.com/mudler/vllm.cpp/issues/81)). What is gated on CPU at k=1..4 is that the propose runs `k-1` draft decode forwards per propose call, that k drafts reach the verify path, and that the drafts DELIVERED to the verify path vary with depth rather than repeating the first one. That last one is counted over a RUN and never per call, because a correct drafter may resample the same token and this fixture does. Two things are NOT gated there. A draft is never accepted at depth, because acceptance is zero at every depth on the synthetic gate model. And nothing here proves the draft at depth j came from the j-th forward. Both are owed to the GPU gate, which must close the second by comparing the per-depth acceptance RATE against a PADDED control rather than by asserting a non-zero acceptance count, because a padded drafter earns acceptance at depth whenever the target's own greedy continuation repeats a token. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). The DRAFT is classified from its own `config.json` rather than from the method string: `Qwen3DSparkModel`, `Gemma4DSparkModel`, and — BEYOND-PIN, mirroring [vllm#52197](https://github.com/vllm-project/vllm/pull/52197) merged 2026-08-17 — `DSparkDraftModel` together with `model_type` `qwen3` all route to the Qwen3 DSpark lane, and every other DSpark draft that DECLARES an architecture is the DeepSeek-V4 variant, which is refused by name because this engine carries only a stub for it (`SPEC-DSPARK-QWEN3-ROUTING`, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)). A draft config carrying no `architectures` key at all is not classified and loads as before, because an absent key is not evidence of a lane. Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). The object is admitted key by key and NOTHING is dropped ([#1160](https://github.com/mudler/vllm.cpp/issues/1160)): the honoured keys are `method`, `num_speculative_tokens`, `model`, `prompt_lookup_min` and `prompt_lookup_max`, plus `draft_sample_method` and `rejection_sample_method` at their upstream defaults `greedy` and `standard`, which are what this engine implements. Any other value of those two names row `SPEC-ACCEPT-VARIANTS` and is refused. A name vLLM's `SpeculativeConfig` declares but this engine does not implement, such as `quantization`, is refused as exactly that, and any other name is refused as unknown with the accepted list. Before this the extra key was discarded, so `draft_sample_method=probabilistic` ran GREEDY and a misspelled `num_speculatve_tokens` took the default, both silently and both at exit 0. For `dspark`, `num_speculative_tokens` may no longer sit BELOW the draft checkpoint's block: DSpark drafts a block, our block is sized from this value alone, and a shorter one drafted a structurally wrong block in silence. It is refused now, before any weight is loaded, naming the block, the config key the block was read from, and the value given ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)). The block is read from the draft config's `dspark_block_size`, or from `block_size` when that key is absent, which is the case on every published Qwen3 draft (`deepseek-ai/dspark_qwen3_4b_block7` and `RadixArk/Qwen3.8-27B-DSpark` both carry `block_size: 7`, so k must be at least 7). vLLM reads only the first key and accepts the shorter value. vLLM also builds its model config BEFORE its speculative config, so a command that names both a target directory it cannot open and a short `k` hears about the target there and about the `k` here. Those are the two recorded divergences, both argued in `.agents/specs/dspark-block-size-guard.md`. A k at or above the block behaves exactly as before. For `dflash`, the DRAFT is likewise classified from its own `config.json`, and a draft that declares `DFlash2DraftModel` is REFUSED at startup, before any weight is read, naming both mechanisms this engine does not implement yet: the grouped dynamic depthwise convolution and the candidate selector (`SPEC-DFLASH2`, [#1314](https://github.com/mudler/vllm.cpp/issues/1314)). It is refused rather than loaded because a DFlash2 checkpoint carries DFlash1's whole tensor set, so the DFlash1 lane would load it with nothing missing and draft worse tokens with no visible symptom: the verify is lossless, so the emitted tokens stay the target's and only acceptance falls. A `DFlashDraftModel` draft is unaffected. A GGUF drafter is classified the same way but by its METADATA, because a GGUF declares no architectures and the published DFlash2 GGUF writes the same `dflash` architecture a DFlash1 one does: a file carrying `dflash.selector_rank`, `dflash.selector_top_k` or `dflash.conv_kernel_size` is refused, and a DFlash1 GGUF, which carries none of them, loads as before. A draft config may also carry a top-level `is_causal`, which now decides every layer's causality ahead of `dflash_config.causal` and ahead of the `layer_types` default, mirroring [vllm#52816](https://github.com/vllm-project/vllm/pull/52816); no published DFlash1 checkpoint declares the key, so their behaviour is unchanged. In a GGUF the same value arrives as `dflash.attention.causal` and is resolved identically. Either spelling is honoured whenever it is DECLARED, as a boolean or as a number, so `"is_causal": 0` means non-causal rather than falling through to the default; a value of any other type is now refused by name instead of being dropped, and the two containers answer alike. When NEITHER explicit key is present, a layer is causal only if its own declared `layer_types` entry is `sliding_attention`. `dflash_config.use_swa` moves the sliding WINDOW onto every layer and no longer makes any layer causal, which is what upstream does ([#1366](https://github.com/mudler/vllm.cpp/issues/1366)); such a draft previously ran every layer causal here and non-causal in vLLM, which cost acceptance and changed no emitted token, so nothing surfaced it. **No checkpoint reaches that arm here yet**, so it changes nothing you can run today: every published DFlash draft that declares `layer_types` also declares no `use_swa`, and the one published draft of the governed shape, `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash`, declares no `layer_types` at all — which this engine's draft-config builder requires, so it fails with the raw `key 'layer_types' not found` before any causality is resolved — while its target architecture `MiMoV2ForCausalLM` is not one this engine serves. A GGUF drafter cannot declare `use_swa` at all. The rule is therefore correct and INERT, and both halves of the gap are owed by `SPEC-DFLASH2` W2 (`.agents/specs/dflash2-spec-decode.md` `## Owed` O4). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | +| `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. For `mtp`, `num_speculative_tokens` sets the draft DEPTH and defaults to the checkpoint's `mtp_num_hidden_layers`, which is 1 on both gate checkpoints, so the default is unchanged. A value above it must be a multiple of it, mirroring vLLM. Depth cannot move the emitted tokens under greedy decoding, and no speed number is claimed above k=1 yet ([#81](https://github.com/mudler/vllm.cpp/issues/81)). What is gated on CPU at k=1..4 is that the propose runs `k-1` draft decode forwards per propose call, that k drafts reach the verify path, and that the drafts DELIVERED to the verify path vary with depth rather than repeating the first one. That last one is counted over a RUN and never per call, because a correct drafter may resample the same token and this fixture does. Two things are NOT gated there. A draft is never accepted at depth, because acceptance is zero at every depth on the synthetic gate model. And nothing here proves the draft at depth j came from the j-th forward. Both are owed to the GPU gate, which must close the second by comparing the per-depth acceptance RATE against a PADDED control rather than by asserting a non-zero acceptance count, because a padded drafter earns acceptance at depth whenever the target's own greedy continuation repeats a token. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). The DRAFT is classified from its own `config.json` rather than from the method string: `Qwen3DSparkModel`, `Gemma4DSparkModel`, and — BEYOND-PIN, mirroring [vllm#52197](https://github.com/vllm-project/vllm/pull/52197) merged 2026-08-17 — `DSparkDraftModel` together with `model_type` `qwen3` all route to the Qwen3 DSpark lane, and every other DSpark draft that DECLARES an architecture is the DeepSeek-V4 variant, which is refused by name because this engine carries only a stub for it (`SPEC-DSPARK-QWEN3-ROUTING`, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)). A draft config carrying no `architectures` key at all is not classified and loads as before, because an absent key is not evidence of a lane. Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). The object is admitted key by key and NOTHING is dropped ([#1160](https://github.com/mudler/vllm.cpp/issues/1160)): the honoured keys are `method`, `num_speculative_tokens`, `model`, `prompt_lookup_min` and `prompt_lookup_max`, plus `draft_sample_method` and `rejection_sample_method` at their upstream defaults `greedy` and `standard`, which are what this engine implements. Any other value of those two names row `SPEC-ACCEPT-VARIANTS` and is refused. A name vLLM's `SpeculativeConfig` declares but this engine does not implement, such as `quantization`, is refused as exactly that, and any other name is refused as unknown with the accepted list. Before this the extra key was discarded, so `draft_sample_method=probabilistic` ran GREEDY and a misspelled `num_speculatve_tokens` took the default, both silently and both at exit 0. For `dspark`, `num_speculative_tokens` may no longer sit BELOW the draft checkpoint's block: DSpark drafts a block, our block is sized from this value alone, and a shorter one drafted a structurally wrong block in silence. It is refused now, before any weight is loaded, naming the block, the config key the block was read from, and the value given ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)). The block is read from the draft config's `dspark_block_size`, or from `block_size` when that key is absent, which is the case on every published Qwen3 draft (`deepseek-ai/dspark_qwen3_4b_block7` and `RadixArk/Qwen3.8-27B-DSpark` both carry `block_size: 7`, so k must be at least 7). vLLM reads only the first key and accepts the shorter value. vLLM also builds its model config BEFORE its speculative config, so a command that names both a target directory it cannot open and a short `k` hears about the target there and about the `k` here. Those are the two recorded divergences, both argued in `.agents/specs/dspark-block-size-guard.md`. A k at or above the block behaves exactly as before. For `dflash`, the DRAFT is likewise classified from its own `config.json`. A safetensors draft that declares `DFlash2DraftModel` is ADMITTED as far as its convolution and no further (`SPEC-DFLASH2`, [#1314](https://github.com/mudler/vllm.cpp/issues/1314)): it loads, it runs the grouped dynamic depthwise convolution around every attention and every MLP sublayer of every draft layer, and it is then REFUSED BY NAME at the candidate selector, which this engine does not implement yet. A notice at STARTUP says exactly that, so the refusal at the first generated token is not a surprise. It is refused rather than sampled with the DFlash1 per-slot argmax because that would succeed: the argmax proposes well-formed tokens, the verify is lossless, so the emitted tokens stay the target's and only acceptance falls, which no token gate can see. A `DFlashDraftModel` draft is unaffected. A GGUF DFlash2 drafter is still refused AT STARTUP, because its weight path does not exist yet — the GGUF drafter arm is a later wave. It is classified by its METADATA rather than by an architecture, because a GGUF declares no architectures and the published DFlash2 GGUF writes the same `dflash` architecture a DFlash1 one does: a file carrying `dflash.selector_rank`, `dflash.selector_top_k` or `dflash.conv_kernel_size` is refused, and a DFlash1 GGUF, which carries none of them, loads as before. Two `config.json` shapes that used to fail the draft-config builder outright now parse: a draft that nests `rope_theta` under `rope_parameters` or `block_size` under `dflash_config` (which BOTH published DFlash2 drafts do), and a draft that declares no `layer_types` at all. A draft declaring `dflash_config.attention_sink_bias` is refused by name, because this engine has no attention sink and loading without one would draft worse tokens in silence. A draft config may also carry a top-level `is_causal`, which now decides every layer's causality ahead of `dflash_config.causal` and ahead of the `layer_types` default, mirroring [vllm#52816](https://github.com/vllm-project/vllm/pull/52816); no published DFlash1 checkpoint declares the key, so their behaviour is unchanged. In a GGUF the same value arrives as `dflash.attention.causal` and is resolved identically. Either spelling is honoured whenever it is DECLARED, as a boolean or as a number, so `"is_causal": 0` means non-causal rather than falling through to the default; a value of any other type is now refused by name instead of being dropped, and the two containers answer alike. When NEITHER explicit key is present, a layer is causal only if its own declared `layer_types` entry is `sliding_attention`. `dflash_config.use_swa` moves the sliding WINDOW onto every layer and no longer makes any layer causal, which is what upstream does ([#1366](https://github.com/mudler/vllm.cpp/issues/1366)); such a draft previously ran every layer causal here and non-causal in vLLM, which cost acceptance and changed no emitted token, so nothing surfaced it. **Still no checkpoint reaches that arm here**, so it changes nothing you can run today: every published DFlash draft that declares `layer_types` also declares no `use_swa`. The one published draft of the governed shape, `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash`, declares no `layer_types` at all, and that now PARSES rather than failing with `key 'layer_types' not found` — but its target architecture `MiMoV2ForCausalLM` is still not one this engine serves, so the draft has nothing to head. A GGUF drafter cannot declare `use_swa` at all. The rule is therefore correct and still INERT (`.agents/specs/dflash2-spec-decode.md` `## Owed` O4, whose parse half is discharged). See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | | `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with ``400 At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit.`` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | | `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count, or an unknown option on `image` / `video` / `audio` is refused at startup rather than defaulted. An unknown option on any other modality name is dropped rather than refused, mirroring upstream, whose fallback `BaseDummyOptions` is the one such dataclass without `extra="forbid"`. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | | `--enable-log-requests` / `--disable-log-requests` | on | Log each incoming request. Mirrors vLLM's flag of the same name | diff --git a/include/vllm/model_executor/models/qwen3_dflash.h b/include/vllm/model_executor/models/qwen3_dflash.h index 78023f568..b0aa87a65 100644 --- a/include/vllm/model_executor/models/qwen3_dflash.h +++ b/include/vllm/model_executor/models/qwen3_dflash.h @@ -66,6 +66,32 @@ struct Qwen3DFlashLayerAttnMode { int64_t sliding_window = 0; // >0 for SWA layers; 0 for full layers }; +// SPEC-DFLASH2 W2 (#1314): the grouped dynamic depthwise convolution that wraps +// ONE sublayer of a DFlash2 draft block. EMPTY on a DFlash1 draft, which is what +// `Qwen3DFlashWeights::IsDflash2` reads. +// +// BEYOND-PIN, from `DFlashGroupedConv` (vllm/model_executor/models/qwen3_dflash2.py +// @ vllm-project/vllm#52816 head `19c9351904df4c63042671bc67a866ca48dc7d6f`). +// +// Both tensors are named exactly as the published checkpoint stores them +// (`z-lab/Qwen3.8-27B-DFlash2` @ `50307d4c4cde6860d4eee73e2547cd786fe8e8a4`, +// safetensors header read 2026-08-19): +// layers.N.attention_conv.base_kernel bf16 (2, 2, 5120) +// layers.N.attention_conv.kernel_projection.weight bf16 (1280, 5120) +// and the same pair under `mlp_conv`. +// +// `base_kernel` dim 0 is the SIDE -- 0 = `prepare` (before the sublayer), 1 = +// `finish` (after it) -- and NOT a tap. On this checkpoint `taps` is also 2, so +// the two axes are indistinguishable by shape and only the port note separates +// them. `kernel_projection` maps hidden -> `2 * taps * num_groups` (1280 = +// 2*2*320 at hidden 5120 / conv_group_size 16), i.e. ONE projection of the +// sublayer input carrying BOTH sides' deltas. +struct Qwen3DFlashConvWeights { + OwnedTensor base_kernel; // bf16 [2, taps, H]; dim 0 is the SIDE + OwnedTensor kernel_projection; // bf16 raw-NK [2*taps*num_groups, H], nk + bool Empty() const { return base_kernel.bytes.empty(); } +}; + // One DFlash draft decoder layer: input/post standard RMSNorm + plain Qwen3 // attention (merged qkv, per-head q/k norm, NeoX RoPE) + SwiGLU MLP. Weights are // kept in the on-disk torch-Linear [N=out,K=in] orientation (nk=true) for @@ -80,6 +106,11 @@ struct Qwen3DFlashLayerWeights { OwnedTensor gate_up_proj; // bf16 raw-NK [2*I, H] (rows gate|up), nk OwnedTensor down_proj; // bf16 raw-NK [H, I], nk Qwen3DFlashLayerAttnMode attn_mode; + // SPEC-DFLASH2 W2 (#1314): the two grouped convolutions of a DFlash2 block, + // wrapping the attention and the MLP sublayer respectively. Both EMPTY on a + // DFlash1 draft, and the forward then runs byte-for-byte as before. + Qwen3DFlashConvWeights attention_conv; + Qwen3DFlashConvWeights mlp_conv; }; // Whole DFlash draft weights. The draft owns its OWN embed_tokens and lm_head @@ -99,6 +130,22 @@ struct Qwen3DFlashWeights { int64_t num_taps = 0; // len(target_layer_ids); fc input = H*num_taps int32_t mask_token_id = -1; // dflash_config.mask_token_id (248070 for 27B) int64_t draft_vocab_size = 0; + // SPEC-DFLASH2 W2 (#1314): the conv geometry. `conv_taps` is + // `dflash_config.conv_kernel_size` and is 0 on a DFlash1 draft, which is what + // makes it the DFlash2 discriminator here -- a DFlash1 checkpoint declares + // none of these keys and carries no conv tensor. + int64_t conv_taps = 0; // dflash_config.conv_kernel_size (2 on both drafts) + int64_t conv_group_size = 0; // dflash_config.conv_group_size (16 on both drafts) + // The QUERY block the conv masks its taps against: `1 + num_speculative_tokens`, + // NOT `dflash_config.block_size`. Upstream sizes it from the speculative config + // (`DFlash2Qwen3DecoderLayer.__init__` @ vllm-project/vllm#52816 head + // `19c9351904df4c63042671bc67a866ca48dc7d6f`) and the checkpoint key only + // supplies that value's DEFAULT. `LoadQwen3DFlash` fills it from the config's + // `block_size` so a direct caller has a usable value; the loader OVERWRITES it + // with `1 + k` once the resolved speculative config is known, because a CLI `k` + // that differs from the checkpoint's default must move the conv's block with it. + int64_t conv_block_size = 0; + bool IsDflash2() const { return conv_taps > 0; } }; // Load the z-lab DFlash draft checkpoint. The on-disk names follow vLLM's diff --git a/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h b/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h index 213a302ed..8eeeb8e6a 100644 --- a/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h +++ b/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h @@ -40,6 +40,33 @@ namespace vllm::v1 { +// SPEC-DFLASH2 W2 (#1314) — refuse a DFlash2 draft's CANDIDATE SELECTOR BY NAME, +// after the draft block forward and before anything samples from its logits. +// +// This is the boundary W2 leaves the architecture at. A `DFlash2DraftModel` draft +// now LOADS and its block forward RUNS, grouped dynamic convolution and all +// (vt::DFlashGroupedConv, wrapped around every attention and MLP sublayer). What +// it cannot do is CHOOSE: upstream replaces the independent per-slot argmax with +// a candidate selector -- keep the target head's top-K per slot, score adjacent +// transitions ` + unary[c]`, and walk the best path from +// the verified anchor (`vllm/model_executor/models/qwen3_dflash2.py` +// `CandidateSelector` + `vllm/v1/worker/gpu/spec_decode/dflash2/speculator.py` @ +// vllm-project/vllm#52816 head `19c9351904df4c63042671bc67a866ca48dc7d6f`), and +// none of that exists here. +// +// Falling through to `SampleDflashBlockDrafts` instead would SUCCEED and be +// silent: the per-slot argmax proposes well-formed tokens, the verify is +// lossless, so the engine still emits the target's tokens and only ACCEPTANCE +// falls -- the one defect class no token gate in this repository can see. That is +// why this is a refusal and not a fallback, and why it is placed AFTER the +// forward: the forward is implemented and gated, the choice is not. +// +// Called from the production draft step (`GPUModelRunner::propose_drafts_block`, +// src/vllm/v1/worker/gpu/runner.cpp) and from `DflashProposeBlock` below, which +// are the only two places that turn draft logits into draft tokens. Owed by W3 of +// the row. +void RefuseDflash2CandidateSelector(const Qwen3DFlashWeights& weights); + // Greedy per-request draft pick over the (1+k) block logits — the greedy branch of // DFlash sample_draft (dflash/speculator.py:_generate_draft :259-273 with // temperature 0 => argmax). `block_logits` is the ForwardBlockLogitsWithContext diff --git a/include/vt/ops.h b/include/vt/ops.h index a4142f6af..63a05a3c9 100644 --- a/include/vt/ops.h +++ b/include/vt/ops.h @@ -113,6 +113,7 @@ enum class OpId : uint8_t { kAttentionDenseFa2, kDFlashBlockAttention, kDFlashPagedBlockAttention, + kDFlashGroupedConv, kReshapeAndCache, kConcatAndCacheMla, kMlaDecodeAttention, @@ -759,6 +760,61 @@ struct DFlashPagedBlockAttentionArgs { int64_t block_size = 0; // rows per paged context page (>0) }; +// Arguments for vt::DFlashGroupedConv — the DFlash2 draft's GROUPED DYNAMIC +// DEPTHWISE CONVOLUTION, wrapped around each attention and each MLP sublayer +// (SPEC-DFLASH2 W2, #1314). +// +// BEYOND-PIN. Ported from `_grouped_conv` and `DFlashGroupedConv` +// (vllm/model_executor/models/qwen3_dflash2.py @ vllm-project/vllm#52816 head +// `19c9351904df4c63042671bc67a866ca48dc7d6f`); the parity pin `555967922` does +// not carry the architecture at all and this op does NOT advance it. +// +// The math, in upstream's own terms: +// +// out[i,c] = sum_{t=0..taps-1} (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c] +// +// with `g(c) = c / group_size` the channel's GROUP, and tap `t` contributing only +// where `(i mod block_size) >= t`. That mask is what makes the op a BLOCK +// convolution rather than a sequence one: a proposal position sees the positions +// before it inside its own (1+k) query block and NOTHING across the block +// boundary, which is how a DFlash2 draft gets causal structure without another +// backbone pass. `i` is the GLOBAL row index, exactly as upstream's +// `torch.arange(hidden_states.shape[0])` is, and it is the right one here for the +// same reason it is there: every request's block is contiguous and +// block_size-aligned, so `i mod block_size` IS the intra-block offset. +// +// Upstream computes that mask on a POWER-OF-TWO block as `position & (block-1)` +// and otherwise as `position % block`. Both arms are mirrored, and both are +// gated: the two published DFlash2 checkpoints ship block 8 and block 16 (both +// power-of-two, `z-lab/Qwen3.8-27B-DFlash2` and `z-lab/Muse-Glimmer-30B-DFlash2`) +// and upstream's own reference test parametrises 5 to reach the modulo arm. +// +// `block_size` is `1 + num_speculative_tokens`, NOT `dflash_config.block_size` — +// upstream sizes the conv by the QUERY block (the bonus token plus the mask +// tokens) rather than by the checkpoint key, which only supplies that value's +// default (`DFlash2Qwen3DecoderLayer.__init__` @ that head). +// +// SIDES. `base_kernel` is `[2, taps, hidden]` and dim 0 is the SIDE — 0 = +// `prepare` (before the sublayer), 1 = `finish` (after it) — NOT a tap. One +// projection of the sublayer input produces BOTH sides' deltas +// (`kernel_projection`: hidden -> 2*taps*num_groups), so `coefficients` is the +// same buffer for both calls and `side` selects the half. Passing the whole +// buffer rather than a slice mirrors upstream's `coefficients[:, side]` view +// without materializing a copy of a non-contiguous slice. +// +// ACCUMULATION. Every intermediate is rounded to the tensor dtype after each +// step, because upstream's chain materializes bf16 tensors at each one +// (`base + delta`, `coefficients * blocks`, `output += ...`). This is elementwise +// with no reduction-order freedom, so the CPU reference and the CUDA kernel are +// BIT-IDENTICAL rather than within an envelope, and the gate asserts that. +struct DFlashGroupedConvArgs { + int64_t block_size = 0; // 1 + num_speculative_tokens (the query block) + int64_t taps = 0; // dflash_config.conv_kernel_size + int64_t num_groups = 0; // hidden_size / conv_group_size + int64_t group_size = 0; // dflash_config.conv_group_size + int64_t side = 0; // 0 = prepare, 1 = finish (selects base/coefficient half) +}; + // Backend-neutral local-attention window, matching FlashAttention's // `window_size=(left, right)` convention. The bounds are inclusive distances // from the bottom-right-aligned absolute query position: (W-1, 0) is a causal @@ -1182,6 +1238,8 @@ using DFlashPagedBlockAttentionFn = void (*)(Queue&, Tensor&, const Tensor&, con const Tensor&, const Tensor&, const Tensor&, const Tensor&, const Tensor&, const Tensor&, const DFlashPagedBlockAttentionArgs&); +using DFlashGroupedConvFn = void (*)(Queue&, Tensor&, const Tensor&, const Tensor&, + const Tensor&, const DFlashGroupedConvArgs&); using ReshapeAndCacheFn = void (*)(Queue&, const Tensor&, const Tensor&, Tensor&, Tensor&, const Tensor&); // fp8 KV-cache store (KV-FP8 W1). k_cache/v_cache are 1-byte fp8 (DType::kI8); @@ -2967,6 +3025,18 @@ void DFlashPagedBlockAttention(Queue& q, Tensor& out, const Tensor& query, const Tensor& block_table, const DFlashPagedBlockAttentionArgs& args); +// DFlash2 grouped dynamic depthwise convolution (SPEC-DFLASH2 W2, #1314). See +// DFlashGroupedConvArgs for the contract and the upstream anchor. Tensors: +// out [T, H] the convolved sublayer stream +// x [T, H] the sublayer input/output stream +// coefficients [T, sides, taps, num_groups] the per-position kernel DELTAS +// base [sides, taps, H] the checkpoint's base_kernel +// with H == num_groups * group_size and `args.side` in [0, sides). All four share +// one float dtype (bf16 on every published checkpoint). CPU is the authoritative +// reference; CUDA mirrors it bit-for-bit. +void DFlashGroupedConv(Queue& q, Tensor& out, const Tensor& x, const Tensor& coefficients, + const Tensor& base, const DFlashGroupedConvArgs& args); + // --- Paged KV-cache write (M1.6). Semantics ported from the FlashAttention // path of vllm/csrc/.../cache_kernels.cu::reshape_and_cache_flash @ e24d1b24; // the NHD cache layout is the one FlashAttentionBackend::get_kv_cache_shape diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index cd4066244..91641764a 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -283,7 +283,20 @@ # of rank-1 factors) rather than how one step is tiled, and it adds three cache # tensors to the MambaSpec. vLLM ships the algorithm for Mamba2 only and cannot # reach GDN (four walls, spec §Upstream chain); SGLang ships the GDN arm. - "KERNEL": (AGENTS / "kernel-matrix.md", 52), + # 53 since 2026-08-19 (#1314): +`KERNEL-DFLASH2-GROUPED-CONV`, the DFlash2 + # draft's grouped DYNAMIC depthwise convolution. A genuinely new family and + # not a variant of `KERNEL-DEPTHWISE-CONV1D`, on all three axes that decide + # a kernel's shape: the weights are DYNAMIC (a per-position delta projected + # from the sublayer input, added to a static per-channel base) rather than + # static, they are GROUPED (one delta per group of channels against one base + # per channel) rather than per-channel, and the tap mask is over the QUERY + # BLOCK (`i mod (1+k)`) rather than causal over the sequence. It also carries + # a SIDE axis no other convolution here has: one projection of the sublayer + # input produces both the prepare-side and the finish-side coefficients. + # Bumped because the row EXISTS, never to make a state transition pass; the + # row is `ACTIVE` rather than `DONE` because its CUDA arm has never compiled + # (spec `## Owed` O6, no `nvcc` on the authoring host). + "KERNEL": (AGENTS / "kernel-matrix.md", 53), # 56 since 2026-07-22: +`BACKEND-ACCEL-PROVIDER` (the acceleration-provider seam # itself, which is a cross-backend platform concern rather than a platform). # 57 since 2026-07-22: +`BACKEND-SEAM-AUDIT` (the accelerator-seam AUDIT — does diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 90c451e79..8832c9a49 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -448,21 +448,38 @@ std::vector ReadDflashDraftArchitectures(const std::string& path) { return architectures; } -// Refuse a DFlash2 draft BY NAME, before any weight is read. +// Classify a DFlash2 draft BY NAME, before any weight is read, and refuse the arm +// that is still missing. // // Upstream selects a different model class and a different speculator on the // `DFlash2DraftModel` architecture (registry.py:628 and // v1/worker/gpu/spec_decode/__init__.py:12-17 @ vllm-project/vllm#52816 head // `19c9351904df4c63042671bc67a866ca48dc7d6f`). This engine selects the draft lane // from the CLI method string alone, and a DFlash2 checkpoint's tensor set is -// DFlash1's PLUS the conv and selector tensors -- so it loads through the DFlash1 -// loader with nothing missing and nothing thrown, and drafts with both new -// mechanisms simply absent. That draft proposes worse tokens, the verify is -// lossless, so the emitted tokens are still the target's and only acceptance -// falls. AGENTS.md requires an unimplemented arm to refuse with the missing part -// named rather than to degrade in silence, and this refusal is what SPEC-DFLASH2 -// W1 ships. -void RefuseDflash2Draft(const std::string& draft_model_path) { +// DFlash1's PLUS the conv and selector tensors -- so without this classification +// it loads through the DFlash1 loader with nothing missing and nothing thrown, +// and drafts with both new mechanisms simply absent. That draft proposes worse +// tokens, the verify is lossless, so the emitted tokens are still the target's +// and only acceptance falls. +// +// SPEC-DFLASH2 W2 (#1314) SPLITS the two container arms, because they are no +// longer in the same state: +// +// * SAFETENSORS is ADMITTED. Its grouped dynamic depthwise convolution is +// implemented (`vt::DFlashGroupedConv`), loaded (`LoadQwen3DFlash` reads the +// per-layer `attention_conv`/`mlp_conv` tensors) and RUN (every layer body of +// `Qwen3DFlashModel`). What is still missing is the candidate selector, and +// that is refused BY NAME one step later, after the conv has executed, at +// `RefuseDflash2CandidateSelector`. Refusing here instead would leave every +// line of W2 unreachable from any production entry point -- AGENTS.md +// `## Nothing lands dead`. The notice below is what a user gets at STARTUP so +// the later refusal is not a surprise; it is a notice and not a warning about +// a degraded result, because there is no degraded result: the engine refuses. +// * GGUF is still REFUSED, because the GGUF drafter ARM is wave W5: neither the +// config reader nor the weight path has a name for a conv tensor, so admitting +// the file would load a DFlash1 draft out of a DFlash2 checkpoint -- the exact +// silent degradation this function exists to prevent. +void CheckDflash2DraftArm(const std::string& draft_model_path) { // WHAT IDENTIFIED THE FILE, which differs by container and is quoted back to // the user because the two arms are otherwise indistinguishable in a message. std::string identity; @@ -484,29 +501,43 @@ void RefuseDflash2Draft(const std::string& draft_model_path) { } identity = "carries the DFlash2-only metadata key \"" + matched + "\""; } else { + // The safetensors arm, ADMITTED as of W2 -- with the boundary stated at + // startup rather than discovered at the first generated token. const std::vector architectures = ReadDflashDraftArchitectures(draft_model_path); if (!vllm::SpeculativeConfig::IsDflash2Draft(architectures)) return; - identity = "declares architecture \"DFlash2DraftModel\""; + std::cerr + << "vllm.cpp: the draft checkpoint at \"" << draft_model_path + << "\" declares architecture \"DFlash2DraftModel\". Its grouped dynamic " + "depthwise convolution is implemented and will run; its CANDIDATE " + "SELECTOR is not implemented, and this draft will be refused by name at " + "its first propose rather than sampled with the DFlash1 per-slot argmax " + "(which would propose worse tokens with no visible symptom, because the " + "verify is lossless). Owed by row SPEC-DFLASH2 wave W3 " + "(.agents/specs/dflash2-spec-decode.md), issue #1314.\n"; + return; } throw std::invalid_argument( "speculative-config: the draft checkpoint at \"" + draft_model_path + "\" " + identity + - ", and the DFlash2 draft lane " - "is not implemented here. Two mechanisms are missing: the grouped dynamic " - "depthwise convolution wrapped around each attention and each MLP sublayer, " - "and the candidate selector that replaces the per-slot argmax with a scored " - "path walk over the target head's top-K " + ", and the DFlash2 GGUF drafter ARM " + "is not implemented here. A GGUF DFlash2 drafter needs its own weight path: " + "`MakeDflashGgufConfig` reads none of the DFlash2 metadata beyond the keys " + "that identify the file, and `LoadQwen3DFlashFromGguf` has no name for the " + "conv or selector tensors. The SAFETENSORS DFlash2 draft is admitted as of " + "SPEC-DFLASH2 W2: its grouped dynamic depthwise convolution is implemented " + "and runs, and it is refused at the candidate selector instead " "(vllm/model_executor/models/qwen3_dflash2.py and " "vllm/v1/worker/gpu/spec_decode/dflash2/speculator.py @ " "vllm-project/vllm#52816 head 19c9351904df4c63042671bc67a866ca48dc7d6f). " - "Loading it through the DFlash1 lane instead would succeed, because a " - "DFlash2 checkpoint carries DFlash1's whole tensor set, and it would draft " - "worse tokens with no visible symptom: the verify is lossless, so the " - "emitted tokens are still the target's and only acceptance falls. Owed by " - "row SPEC-DFLASH2 (.agents/specs/dflash2-spec-decode.md), issue #1314 " + "Loading this file through the DFlash1 GGUF lane instead would succeed, " + "because a DFlash2 GGUF carries DFlash1's whole tensor set and declares the " + "same `general.architecture`, and it would draft worse tokens with no " + "visible symptom: the verify is lossless, so the emitted tokens are still " + "the target's and only acceptance falls. Owed by row SPEC-DFLASH2 wave W5 " + "(.agents/specs/dflash2-spec-decode.md), issue #1314 " "(https://github.com/mudler/vllm.cpp/issues/1314). Use a DFlashDraftModel " - "checkpoint until that row lands."); + "checkpoint until that wave lands."); } // SPEC-DSPARK-QWEN3-ROUTING (#1193): the two keys upstream classifies a DSpark @@ -786,6 +817,24 @@ std::unique_ptr LoadDflashDraft( if (draft->config.vocab_size == 0) { draft->config.vocab_size = draft->weights.embed_tokens.shape[0]; } + // SPEC-DFLASH2 W2 (#1314): the conv's block is the QUERY block, and the + // resolved `k` is its authority. `LoadQwen3DFlash` seeded it from the + // checkpoint's own `block_size`, which is only that value's DEFAULT; a CLI + // `--speculative-config` that names a different k must move the conv's tap mask + // with it, exactly as upstream sizes the conv from + // `1 + speculative_config.num_speculative_tokens` and never from the config key + // (`DFlash2Qwen3DecoderLayer.__init__` @ vllm-project/vllm#52816 head + // `19c9351904df4c63042671bc67a866ca48dc7d6f`). A conv masking against the wrong + // block is acceptance-only and token-invisible, so it is set from one place. + if (draft->weights.IsDflash2()) { + draft->weights.conv_block_size = draft->k + 1; + std::cerr << "vllm.cpp: DFlash2 draft: grouped conv taps=" + << draft->weights.conv_taps + << " group=" << draft->weights.conv_group_size + << " block=" << draft->weights.conv_block_size + << "; the candidate selector is NOT implemented and this draft will " + "be refused by name at its first propose (SPEC-DFLASH2 W3, #1314)\n"; + } std::cerr << "vllm.cpp: DFlash draft loaded from " << source_kind << " " << draft_dir << " (k=" << draft->k << ", taps=" << num_taps << ", mask=" << mask_id << ", vocab=" << draft->config.vocab_size @@ -1107,7 +1156,7 @@ std::optional LoadedEngine::ResolveSpecConfig( // OMISSION rather than by decision. This is the production caller // `SpeculativeConfig::IsDflash2Draft` would otherwise lack. if (cli.draft_model_path.has_value()) { - RefuseDflash2Draft(*cli.draft_model_path); + CheckDflash2DraftArm(*cli.draft_model_path); } vllm::SpeculativeConfig cfg = vllm::SpeculativeConfig::ResolveDflash(*cli.num_speculative_tokens); @@ -1807,13 +1856,13 @@ std::unique_ptr LoadedEngine::FromModelDir( // checkpoint had already been read through the DFlash1 loader. Both target // containers pass through this line, and the `.gguf` branch immediately below // returns before the later one. This is a REFUSAL and not a second resolution: - // it calls the same `RefuseDflash2Draft` the constructor's `ResolveSpecConfig` + // it calls the same `CheckDflash2DraftArm` the constructor's `ResolveSpecConfig` // calls and decides nothing else, so the classification keeps one owner and // one message. if (params.speculative_config.has_value() && params.speculative_config->method == "dflash" && params.speculative_config->draft_model_path.has_value()) { - RefuseDflash2Draft(*params.speculative_config->draft_model_path); + CheckDflash2DraftArm(*params.speculative_config->draft_model_path); } // A single `.gguf` file: config + weights + tokenizer all come from the diff --git a/src/vllm/model_executor/models/qwen3_dflash.cpp b/src/vllm/model_executor/models/qwen3_dflash.cpp index b16db7ec6..097f73261 100644 --- a/src/vllm/model_executor/models/qwen3_dflash.cpp +++ b/src/vllm/model_executor/models/qwen3_dflash.cpp @@ -34,6 +34,103 @@ using namespace dense_attn; // Dev, DBuf, ResidentWeight, Reshape, MakeRopeArgs constexpr int64_t kPadSlotId = -1; // vLLM PAD_SLOT_ID (attention/backends/utils.py:45) +// --------------------------------------------------------------------------- +// SPEC-DFLASH2 W2 (#1314) — the grouped dynamic depthwise convolution that wraps +// each attention and each MLP sublayer of a DFlash2 draft block. +// +// BEYOND-PIN, from `DFlashGroupedConv.prepare` / `.finish` and +// `DFlash2Qwen3DecoderLayer.forward` +// (vllm/model_executor/models/qwen3_dflash2.py @ vllm-project/vllm#52816 head +// `19c9351904df4c63042671bc67a866ca48dc7d6f`). Upstream's decoder layer is: +// +// hidden, coefficients = self.attention_conv.prepare(hidden) +// hidden = self.self_attn(positions, hidden) +// hidden = self.attention_conv.finish(hidden, coefficients) +// hidden, residual = self.post_attention_layernorm(hidden, residual) +// hidden, coefficients = self.mlp_conv.prepare(hidden) +// hidden = self.mlp(hidden) +// hidden = self.mlp_conv.finish(hidden, coefficients) +// +// TWO things about the shape of this pair are load-bearing and neither is +// visible in a token gate, because a DFlash2 draft whose conv is wrong still +// emits the TARGET's tokens (the verify is lossless) and loses only acceptance: +// +// * ONE projection, TWO sides. `prepare` projects the sublayer INPUT once into +// `[T, 2, taps, num_groups]` and convolves with side 0; `finish` reuses that +// SAME buffer with side 1, over the sublayer OUTPUT. So the finish +// coefficients are a function of the input, not of the output, and computing +// them again after the sublayer would be a different model. +// * The conv's block is the QUERY block, `1 + k`, and the taps are zeroed +// across its boundary, which is what `weights.conv_block_size` carries. +// +// `stream` is convolved IN PLACE (the DBuf is replaced by the conv output), +// mirroring upstream's rebinding of `hidden_states`. +DBuf DflashConvPrepare(Dev d, const Qwen3DFlashConvWeights& cw, + const Qwen3DFlashWeights& weights, const HfConfig& config, + DBuf* stream) { + const int64_t T = stream->t().shape[0]; + const int64_t H = config.hidden_size; + const int64_t taps = weights.conv_taps; + const int64_t groups = H / weights.conv_group_size; + // ONE projection of the sublayer input -> [T, 2, taps, num_groups]. The GEMM + // writes a flat [T, 2*taps*num_groups] view of the same buffer, which is the + // reshape upstream spells as `.reshape(hidden.shape[0], 2, taps, num_groups)`. + DBuf coef(d, DType::kBF16, {T, 2, taps, groups}); + { + Tensor flat = Reshape(coef.t(), {T, 2 * taps * groups}); + Tensor wp = ResidentWeight(d, cw.kernel_projection); + vt::MatmulBT(d.q, flat, stream->t(), wp); + } + DBuf out(d, DType::kBF16, {T, H}); + Tensor base = ResidentWeight(d, cw.base_kernel, {2, taps, H}); + vt::DFlashGroupedConvArgs a; + a.block_size = weights.conv_block_size; + a.taps = taps; + a.num_groups = groups; + a.group_size = weights.conv_group_size; + a.side = 0; // prepare + vt::DFlashGroupedConv(d.q, out.t(), stream->t(), coef.t(), base, a); + *stream = std::move(out); + return coef; +} + +void DflashConvFinish(Dev d, const Qwen3DFlashConvWeights& cw, + const Qwen3DFlashWeights& weights, const HfConfig& config, + DBuf* stream, const DBuf& coef) { + const int64_t T = stream->t().shape[0]; + const int64_t H = config.hidden_size; + const int64_t taps = weights.conv_taps; + const int64_t groups = H / weights.conv_group_size; + DBuf out(d, DType::kBF16, {T, H}); + Tensor base = ResidentWeight(d, cw.base_kernel, {2, taps, H}); + vt::DFlashGroupedConvArgs a; + a.block_size = weights.conv_block_size; + a.taps = taps; + a.num_groups = groups; + a.group_size = weights.conv_group_size; + a.side = 1; // finish, reading the SAME coefficients the prepare projected + vt::DFlashGroupedConv(d.q, out.t(), stream->t(), coef.t(), base, a); + *stream = std::move(out); +} + +// The conv masks its taps by `row index mod conv_block_size`, exactly as +// upstream's `torch.arange(hidden_states.shape[0]) % block_size` does. That is +// the intra-block offset ONLY while every request block is contiguous and +// `conv_block_size`-aligned, which is the uniform (1+k) DFlash batch. A ragged +// batch would silently mask the wrong taps -- acceptance-only and token-invisible +// -- so it is refused here rather than discovered on a gate host. +void CheckDflashConvBatch(const Qwen3DFlashWeights& weights, const std::vector& cu) { + VT_CHECK(weights.conv_block_size > 0, + "qwen3_dflash2: conv_block_size must be set (1 + num_speculative_tokens)"); + for (size_t r = 0; r + 1 < cu.size(); ++r) { + VT_CHECK(cu[r] % weights.conv_block_size == 0 && + cu[r + 1] - cu[r] == static_cast(weights.conv_block_size), + "qwen3_dflash2: the grouped convolution needs a uniform " + "conv_block_size-aligned query block per request"); + } +} + + // Device-resident per-layer context K/V (D7). The D5 path downloaded each layer's // projected K/V to host (2 D->H copies/layer) and re-uploaded them in the block // forward's [context;block] host interleave. This helper keeps the projected K/V @@ -240,6 +337,7 @@ std::vector Qwen3DFlashModel::ForwardBlockLogits( "qwen3_dflash: cu_seqlens must span [0,T]"); VT_CHECK(weights.layers.size() == static_cast(config.num_hidden_layers), "qwen3_dflash: one layer weight per config.num_hidden_layers"); + if (weights.IsDflash2()) CheckDflashConvBatch(weights, cu); // Embed: hidden[T,H] bf16 = embed_tokens[input_ids]; mask slots take // embed_tokens[mask_token_id] naturally (in-vocab), or the dedicated mask @@ -287,6 +385,11 @@ std::vector Qwen3DFlashModel::ForwardBlockLogits( else vt::RmsNorm(d.q, dhn.t(), hidden.t(), w_in, vt::RmsNormArgs{eps, false}, &res.t()); + // SPEC-DFLASH2 W2 (#1314): attention_conv.prepare, before the sublayer. + DBuf attn_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + attn_coef = DflashConvPrepare(d, layer.attention_conv, weights, config, &dhn); + // attention over the context-free block (routes through DFlashBlockAttention). // Reuse the block helper but feed the real positions to RoPE. DBuf attn = [&]() -> DBuf { @@ -336,6 +439,11 @@ std::vector Qwen3DFlashModel::ForwardBlockLogits( return o; }(); + // SPEC-DFLASH2 W2: attention_conv.finish, over the sublayer OUTPUT with the + // coefficients the prepare projected off the sublayer INPUT. + if (weights.IsDflash2()) + DflashConvFinish(d, layer.attention_conv, weights, config, &attn, attn_coef); + // post_attention_layernorm (std add+RMSNorm). Tensor w_post = ResidentWeight(d, layer.post_attention_layernorm, {H}); DBuf dh2(d, DType::kBF16, {T, H}); @@ -349,11 +457,17 @@ std::vector Qwen3DFlashModel::ForwardBlockLogits( // — byte-for-byte the same op sequence the inline path ran, now on the same // exemplar as qwen3.cpp MlpBlock. (Tier-A1 fold, arch-fusion-fold-plan.) const int64_t I = config.intermediate_size; + // SPEC-DFLASH2 W2: mlp_conv.prepare / .finish around the MLP sublayer. + DBuf mlp_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + mlp_coef = DflashConvPrepare(d, layer.mlp_conv, weights, config, &dh2); DBuf act = layers::UnquantizedMlpGateUpMethod(&layer.gate_up_proj, I).Apply(d, dh2.t()); Tensor wdn = ResidentWeight(d, layer.down_proj); DBuf down(d, DType::kBF16, {T, H}); vt::MatmulBT(d.q, down.t(), act.t(), wdn); + if (weights.IsDflash2()) + DflashConvFinish(d, layer.mlp_conv, weights, config, &down, mlp_coef); if (per_layer_out != nullptr) { DBuf tmp(d, DType::kF32, {T, H}); vt::CastF32(d.q, tmp.t(), down.t()); @@ -455,6 +569,7 @@ static std::vector ForwardWithCtxKVDev( const int64_t C = ckv.num_ctx; VT_CHECK(ctx_cu.back() == static_cast(C), "ForwardWithCtxKVDev: ctx_cu.back() must equal num_ctx"); + if (weights.IsDflash2()) CheckDflashConvBatch(weights, cu); // Combined [context; block] per-request layout for the attention (cu_comb), plus // the DEVICE index maps (D7) that place context/block rows into the combined @@ -527,6 +642,11 @@ static std::vector ForwardWithCtxKVDev( else vt::RmsNorm(d.q, dhn.t(), hidden.t(), w_in, vt::RmsNormArgs{eps, false}, &res.t()); + // SPEC-DFLASH2 W2 (#1314): attention_conv.prepare, before the sublayer. + DBuf attn_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + attn_coef = DflashConvPrepare(d, layer.attention_conv, weights, config, &dhn); + // Block q/k/v: same per-layer path as the context-free forward. const float scale = 1.0F / std::sqrt(static_cast(Dh)); DBuf q(d, DType::kBF16, {Tq, qdim}); @@ -590,6 +710,11 @@ static std::vector ForwardWithCtxKVDev( DBuf attn(d, DType::kBF16, {Tq, H}); vt::MatmulBT(d.q, attn.t(), a.t(), wo); + // SPEC-DFLASH2 W2 (#1314): attention_conv.finish. Its prepare ran above, on + // the input_layernorm output, before the qkv projection. + if (weights.IsDflash2()) + DflashConvFinish(d, layer.attention_conv, weights, config, &attn, attn_coef); + // post_attention_layernorm + SwiGLU MLP (unchanged from ForwardBlockLogits). Tensor w_post = ResidentWeight(d, layer.post_attention_layernorm, {H}); DBuf dh2(d, DType::kBF16, {Tq, H}); @@ -598,6 +723,9 @@ static std::vector ForwardWithCtxKVDev( else vt::RmsNorm(d.q, dh2.t(), attn.t(), w_post, vt::RmsNormArgs{eps, false}, &res.t()); const int64_t I = config.intermediate_size; + DBuf mlp_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + mlp_coef = DflashConvPrepare(d, layer.mlp_conv, weights, config, &dh2); Tensor wgu = ResidentWeight(d, layer.gate_up_proj); DBuf gu(d, DType::kBF16, {Tq, 2 * I}); vt::MatmulBT(d.q, gu.t(), dh2.t(), wgu); @@ -606,6 +734,8 @@ static std::vector ForwardWithCtxKVDev( Tensor wdn = ResidentWeight(d, layer.down_proj); DBuf down(d, DType::kBF16, {Tq, H}); vt::MatmulBT(d.q, down.t(), act.t(), wdn); + if (weights.IsDflash2()) + DflashConvFinish(d, layer.mlp_conv, weights, config, &down, mlp_coef); if (per_layer_out != nullptr) { DBuf tmp(d, DType::kF32, {Tq, H}); vt::CastF32(d.q, tmp.t(), down.t()); @@ -902,6 +1032,10 @@ static DBuf ForwardPagedBody(Dev d, const DflashDeviceKVStore& store, const Tens const int64_t qdim = Hq * Dh, kdim = Hkv * Dh; const int64_t vocab = weights.draft_vocab_size; const float eps = static_cast(config.rms_norm_eps); + // SPEC-DFLASH2 W2 (#1314): this body serves ONE request whose whole (1+k) + // query block is rows [0, Tq), so the conv's alignment condition is just + // Tq == conv_block_size. + if (weights.IsDflash2()) CheckDflashConvBatch(weights, {0, static_cast(Tq)}); Tensor cur = hidden_in; std::vector keep; // keep each layer's post-MLP `down` alive across iterations keep.reserve(static_cast(config.num_hidden_layers)); @@ -916,6 +1050,14 @@ static DBuf ForwardPagedBody(Dev d, const DflashDeviceKVStore& store, const Tens else vt::RmsNorm(d.q, dhn.t(), cur, w_in, vt::RmsNormArgs{eps, false}, &res.t()); + // SPEC-DFLASH2 W2 (#1314): attention_conv.prepare. This body is the one the + // production decode path reaches (runner.cpp -> ForwardBlockLogitsWithDeviceKV) + // and the one that is CUDA-graph captured; vt::DFlashGroupedConv is a plain + // stream kernel with no host upload, so it captures like every other op here. + DBuf attn_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + attn_coef = DflashConvPrepare(d, layer.attention_conv, weights, config, &dhn); + const float scale = 1.0F / std::sqrt(static_cast(Dh)); DBuf q(d, DType::kBF16, {Tq, qdim}); DBuf k(d, DType::kBF16, {Tq, kdim}); @@ -950,6 +1092,10 @@ static DBuf ForwardPagedBody(Dev d, const DflashDeviceKVStore& store, const Tens DBuf attn(d, DType::kBF16, {Tq, H}); vt::MatmulBT(d.q, attn.t(), a, wo); + // SPEC-DFLASH2 W2: attention_conv.finish. + if (weights.IsDflash2()) + DflashConvFinish(d, layer.attention_conv, weights, config, &attn, attn_coef); + Tensor w_post = ResidentWeight(d, layer.post_attention_layernorm, {H}); DBuf dh2(d, DType::kBF16, {Tq, H}); if (FusedChainAdoptEnabled()) @@ -957,6 +1103,9 @@ static DBuf ForwardPagedBody(Dev d, const DflashDeviceKVStore& store, const Tens else vt::RmsNorm(d.q, dh2.t(), attn.t(), w_post, vt::RmsNormArgs{eps, false}, &res.t()); const int64_t I = config.intermediate_size; + DBuf mlp_coef(d, DType::kBF16, {0}); + if (weights.IsDflash2()) + mlp_coef = DflashConvPrepare(d, layer.mlp_conv, weights, config, &dh2); Tensor wgu = ResidentWeight(d, layer.gate_up_proj); DBuf gu(d, DType::kBF16, {Tq, 2 * I}); vt::MatmulBT(d.q, gu.t(), dh2.t(), wgu); @@ -965,6 +1114,8 @@ static DBuf ForwardPagedBody(Dev d, const DflashDeviceKVStore& store, const Tens Tensor wdn = ResidentWeight(d, layer.down_proj); DBuf down(d, DType::kBF16, {Tq, H}); vt::MatmulBT(d.q, down.t(), act.t(), wdn); + if (weights.IsDflash2()) + DflashConvFinish(d, layer.mlp_conv, weights, config, &down, mlp_coef); keep.push_back(std::move(down)); cur = keep.back().t(); } diff --git a/src/vllm/model_executor/models/qwen3_dflash_weights.cpp b/src/vllm/model_executor/models/qwen3_dflash_weights.cpp index 9c507ca32..b7c4e2554 100644 --- a/src/vllm/model_executor/models/qwen3_dflash_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_dflash_weights.cpp @@ -91,6 +91,15 @@ OwnedTensor ConcatRawNK(const TensorResolver& get, const std::vector(); cfg.head_dim = c.at("head_dim").get(); cfg.rotary_dim = cfg.head_dim; - cfg.rope_theta = c.at("rope_theta").get(); + // ROPE THETA, in BOTH spellings (SPEC-DFLASH2 W2, spec `## Owed` O3, #1314). + // `transformers` 5 moved the RoPE settings under `rope_parameters`, and BOTH + // published DFlash2 drafts nest it there and declare NO top-level `rope_theta` + // -- so the flat `c.at("rope_theta")` threw before any DFlash2 mechanism could + // be reached. Upstream reads one resolved `config.rope_parameters` + // (`qwen3_dflash.py:340` @ vllm-project/vllm#52816 head + // `19c9351904df4c63042671bc67a866ca48dc7d6f`) after + // `set_default_rope_theta(config, default_theta=1000000)` (`:304`), which is + // where the fallback value below comes from. This is a FALLBACK and not a + // replacement: every published DFlash1 draft carries the flat spelling and + // must keep taking it, which is why the flat key is tested FIRST. + cfg.rope_theta = kDflashDefaultRopeTheta; + if (c.contains("rope_theta") && c.at("rope_theta").is_number()) { + cfg.rope_theta = c.at("rope_theta").get(); + } else if (c.contains("rope_parameters") && c.at("rope_parameters").is_object() && + c.at("rope_parameters").contains("rope_theta") && + c.at("rope_parameters").at("rope_theta").is_number()) { + cfg.rope_theta = c.at("rope_parameters").at("rope_theta").get(); + } cfg.intermediate_size = c.at("intermediate_size").get(); cfg.vocab_size = c.at("vocab_size").get(); cfg.num_hidden_layers = c.at("num_hidden_layers").get(); cfg.rms_norm_eps = c.at("rms_norm_eps").get(); cfg.sliding_window = c.at("sliding_window").get(); - cfg.layer_types = c.at("layer_types").get>(); + // LAYER TYPES are OPTIONAL (spec `## Owed` O4, #1314, #1366). Upstream reads + // `getattr(config, "layer_types", None)` (`qwen3_dflash.py:134`, and `:66` in + // `_dflash_layer_causal`, @ that head), so an absent key is upstream's `None` + // and an EMPTY vector here -- which is exactly the state + // `ResolveQwen3DFlashAttnModes` already treats as "no declared layer types", + // taking `dflash_config.use_swa` for the window and NON-causal for the + // causality. `c.at("layer_types")` instead threw + // `[json.exception.out_of_range.403]` on the only published draft of that + // shape (`XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash`), which is why #1366's + // `use_swa` repair was UNREACHED at its own merge commit. + if (c.contains("layer_types") && c.at("layer_types").is_array()) { + cfg.layer_types = c.at("layer_types").get>(); + } cfg.raw = nlohmann::json::object(); cfg.raw["dflash_config"] = c.at("dflash_config"); - cfg.raw["block_size"] = c.at("block_size"); + const nlohmann::json& dflash_cfg = cfg.raw.at("dflash_config"); + // ATTENTION SINK BIAS is REFUSED BY NAME (spec `## Owed` O4, #1314). + // + // Upstream reads `dflash_config.attention_sink_bias`, falling back to a + // top-level `add_swa_attention_sink_bias`, and when it is truthy it allocates a + // per-head `attention_sink_bias` parameter and passes it into its `Attention` + // as `sinks=` (`qwen3_dflash.py:309-313` and `:240-257` @ that head). This lane + // has NO attention sink of any kind: `vt::DFlashBlockAttention` and its paged + // sibling compute a plain max-subtracted softmax with no extra denominator + // term, and the loader has no name for the tensor. + // + // Refusing rather than ignoring is the whole point of this arm. The key sits on + // `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash`, which is ALSO the only published draft + // that declares no `layer_types` -- so before the fallback above, that + // checkpoint died on a loud `key 'layer_types' not found`, and the fallback + // alone would have converted that loud failure into a draft that loads with the + // sinks silently absent. A missing sink moves acceptance and nothing else: the + // verify is lossless, so the emitted tokens are still the target's and no token + // gate can see it. That is the exact defect class this row exists to remove. + // + // A FALSY value is upstream's own default and is not refused, because upstream + // then creates no sink parameter at all and the two engines agree. + const auto sink_declared = [](const nlohmann::json& obj, const char* key) { + if (!obj.is_object() || !obj.contains(key)) return false; + const nlohmann::json& v = obj.at(key); + if (v.is_boolean()) return v.get(); + if (v.is_number()) return v.get() != 0.0; + return !v.is_null(); + }; + VT_CHECK(!sink_declared(dflash_cfg, "attention_sink_bias") && + !sink_declared(c, "add_swa_attention_sink_bias"), + "qwen3_dflash: this draft declares a per-head attention sink " + "(dflash_config.attention_sink_bias / add_swa_attention_sink_bias), and " + "this engine has no attention sink: vt::DFlashBlockAttention computes a " + "plain softmax with no sink term. Upstream allocates the parameter and " + "passes it as Attention(sinks=...) " + "(vllm/model_executor/models/qwen3_dflash.py:240-257,309-313 @ " + "vllm-project/vllm#52816 head 19c9351904df4c63042671bc67a866ca48dc7d6f). " + "Loading without it would succeed and draft worse tokens with no visible " + "symptom, because the verify is lossless and only acceptance falls. Owed " + "by row SPEC-DFLASH2 (.agents/specs/dflash2-spec-decode.md `## Owed` O4), " + "issue #1314 (https://github.com/mudler/vllm.cpp/issues/1314)."); + // BLOCK SIZE, in BOTH spellings (spec `## Owed` O3). The DFlash1 drafts declare + // it at the top level; both published DFlash2 drafts declare it ONLY as + // `dflash_config.block_size`, so the flat read threw on them too. Upstream + // never reads a top-level `block_size` in this file at all -- the conv's block + // is `1 + speculative_config.num_speculative_tokens` + // (`qwen3_dflash2.py` `DFlash2Qwen3DecoderLayer.__init__` @ that head) and the + // checkpoint key only supplies that value's DEFAULT, which this engine's loader + // resolves from the CLI. The key is carried for the DFlash1 callers that read + // it and is not the conv's authority. + if (c.contains("block_size")) { + cfg.raw["block_size"] = c.at("block_size"); + } else if (dflash_cfg.contains("block_size")) { + cfg.raw["block_size"] = dflash_cfg.at("block_size"); + } // SPEC-DFLASH2 W1 (#1314): the top-level attention semantics, which // ResolveQwen3DFlashAttnModes resolves ahead of every legacy arm. Upstream gets // this key for free by reading it off a HuggingFace config object @@ -273,6 +367,46 @@ Qwen3DFlashWeights LoadQwen3DFlash(const TensorResolver& get, const HfConfig& co out.mask_token_id = mask_token_id; out.draft_vocab_size = config.vocab_size; + // SPEC-DFLASH2 W2 (#1314): the grouped-convolution geometry, read off the + // draft's own dflash_config. A DFlash1 checkpoint declares NONE of these keys + // and carries no conv tensor, so `conv_taps` stays 0, `IsDflash2()` is false, + // and everything below this point loads exactly as before -- which is the + // inertness the DFlash1 gates assert. + { + const nlohmann::json empty = nlohmann::json::object(); + const nlohmann::json& dflash = + (config.raw.is_object() && config.raw.contains("dflash_config") && + config.raw.at("dflash_config").is_object()) + ? config.raw.at("dflash_config") + : empty; + const bool has_taps = + dflash.contains("conv_kernel_size") && dflash.at("conv_kernel_size").is_number(); + const bool has_group = + dflash.contains("conv_group_size") && dflash.at("conv_group_size").is_number(); + // Both or neither. A checkpoint declaring one alone is not a shape this port + // knows how to size, and guessing the other would size the projection wrong + // and be invisible: the draft would still emit the target's tokens. + VT_CHECK(has_taps == has_group, + "qwen3_dflash: dflash_config declares only one of conv_kernel_size / " + "conv_group_size; a DFlash2 draft declares both (SPEC-DFLASH2, #1314)"); + if (has_taps) { + out.conv_taps = dflash.at("conv_kernel_size").get(); + out.conv_group_size = dflash.at("conv_group_size").get(); + VT_CHECK(out.conv_taps > 0 && out.conv_group_size > 0, + "qwen3_dflash: conv_kernel_size and conv_group_size must be > 0"); + // Upstream refuses a group size that does not divide hidden + // (`DFlashGroupedConv.__init__` @ the PR head), with the same polarity. + VT_CHECK(config.hidden_size % out.conv_group_size == 0, + "qwen3_dflash: conv_group_size must divide hidden_size"); + // The DEFAULT query block, from the checkpoint. The loader overwrites it + // with `1 + k` once the resolved speculative config is known; see the field + // comment on Qwen3DFlashWeights::conv_block_size. + if (config.raw.contains("block_size") && config.raw.at("block_size").is_number()) { + out.conv_block_size = config.raw.at("block_size").get(); + } + } + } + // embed_tokens + lm_head are SHARED from the target (the draft ckpt omits them, // see TryLoadBf16); load if present, else leave empty for the caller to fill. out.embed_tokens = TryLoadBf16(get, "embed_tokens.weight", /*nk=*/false); @@ -299,6 +433,39 @@ Qwen3DFlashWeights LoadQwen3DFlash(const TensorResolver& get, const HfConfig& co ConcatRawNK(get, {mlp + "gate_proj.weight", mlp + "up_proj.weight"}, "gate_up"); layer.down_proj = LoadBf16RawNK(get, mlp + "down_proj.weight"); layer.attn_mode = modes[static_cast(i)]; + // SPEC-DFLASH2 W2 (#1314): the two grouped convolutions, under the exact + // names the published checkpoint stores them under. Loaded only for a DFlash2 + // draft; a DFlash1 checkpoint has no such tensor and asking for one would + // throw "tensor not found" on every existing drafter. + if (out.IsDflash2()) { + const int64_t groups = config.hidden_size / out.conv_group_size; + for (const char* which : {"attention_conv.", "mlp_conv."}) { + const std::string cp = base + which; + Qwen3DFlashConvWeights conv; + conv.base_kernel = LoadBf16Direct(get, cp + "base_kernel"); + conv.kernel_projection = LoadBf16RawNK(get, cp + "kernel_projection.weight"); + // SHAPES, asserted rather than assumed. `base_kernel` is + // [SIDES=2, taps, H] -- dim 0 is prepare/finish and NOT a tap, and on the + // published 27B draft both are 2, so nothing but this check separates a + // correct load from a transposed one. `kernel_projection` is + // [2*taps*num_groups, H]: one projection of the sublayer input carrying + // BOTH sides' deltas. + VT_CHECK(conv.base_kernel.rank == 3 && conv.base_kernel.shape[0] == 2 && + conv.base_kernel.shape[1] == out.conv_taps && + conv.base_kernel.shape[2] == config.hidden_size, + "qwen3_dflash: " + cp + "base_kernel must be [2, conv_kernel_size, H]"); + VT_CHECK(conv.kernel_projection.rank == 2 && + conv.kernel_projection.shape[0] == 2 * out.conv_taps * groups && + conv.kernel_projection.shape[1] == config.hidden_size, + "qwen3_dflash: " + cp + + "kernel_projection.weight must be [2*conv_kernel_size*num_groups, H]"); + if (std::string(which) == "attention_conv.") { + layer.attention_conv = std::move(conv); + } else { + layer.mlp_conv = std::move(conv); + } + } + } out.layers.push_back(std::move(layer)); } diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index afb82a1bf..502bea3b1 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -2752,6 +2752,14 @@ void GPUModelRunner::propose_drafts_block( Qwen3DFlashModel::ForwardBlockLogitsWithDeviceKV( stores, ctx_cu, blk_ids, blk_pos, blk_cu, backbone, config, queue_); const auto t_fwd1 = std::chrono::steady_clock::now(); + // SPEC-DFLASH2 W2 (#1314): the PRODUCTION boundary of the DFlash2 port. The + // block forward above just ran the draft's grouped dynamic convolution + // (vt::DFlashGroupedConv, wrapped around every attention and MLP sublayer of + // every layer); the candidate selector that must choose from these logits is + // W3 and is refused BY NAME here rather than silently replaced by the DFlash1 + // per-slot argmax `sample` is about to apply. See + // RefuseDflash2CandidateSelector for why a fallback is not admissible. + RefuseDflash2CandidateSelector(backbone); const std::vector> drafts = sample(block_logits, P, anchors); const auto t_smp1 = std::chrono::steady_clock::now(); if (propose_trace) { diff --git a/src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp b/src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp index 0306e9f43..101a9399c 100644 --- a/src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp +++ b/src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp @@ -10,6 +10,28 @@ namespace vllm::v1 { +void RefuseDflash2CandidateSelector(const Qwen3DFlashWeights& weights) { + if (!weights.IsDflash2()) return; + VT_CHECK(false, + "dflash2: this draft is a DFlash2 draft (its dflash_config declares " + "conv_kernel_size/conv_group_size and its layers carry the " + "attention_conv/mlp_conv tensors). Its grouped dynamic depthwise " + "convolution IS implemented and just ran; its CANDIDATE SELECTOR is not. " + "Upstream replaces the per-slot argmax with a scored path walk over the " + "target head's top-K -- CandidateSelector " + "(vllm/model_executor/models/qwen3_dflash2.py) plus the walk kernel and " + "the realized-q draft-logit cache " + "(vllm/v1/worker/gpu/spec_decode/dflash2/speculator.py) @ " + "vllm-project/vllm#52816 head 19c9351904df4c63042671bc67a866ca48dc7d6f. " + "Sampling this block with the DFlash1 per-slot argmax instead would " + "succeed and propose worse tokens with NO visible symptom: the verify is " + "lossless, so the emitted tokens are still the target's and only " + "acceptance falls. Owed by row SPEC-DFLASH2 wave W3 " + "(.agents/specs/dflash2-spec-decode.md), issue #1314 " + "(https://github.com/mudler/vllm.cpp/issues/1314). Use a DFlashDraftModel " + "checkpoint until that wave lands."); +} + std::vector> SampleDflashBlockDrafts( const std::vector& block_logits, int num_reqs, int k, int64_t draft_vocab) { @@ -74,6 +96,9 @@ DflashProposeResult DflashProposeBlock( context_states, context_positions, ctx_cu, block_input_ids, block_positions, block_cu, weights, config, queue); + // SPEC-DFLASH2 W2 (#1314): the conv has run; the selector has not been ported. + RefuseDflash2CandidateSelector(weights); + DflashProposeResult out; out.draft_token_ids = SampleDflashBlockDrafts(block_logits, num_reqs, k, weights.draft_vocab_size); diff --git a/src/vt/cpu/cpu_ops.cpp b/src/vt/cpu/cpu_ops.cpp index 4768b52eb..0b3959fef 100644 --- a/src/vt/cpu/cpu_ops.cpp +++ b/src/vt/cpu/cpu_ops.cpp @@ -3011,6 +3011,90 @@ void DFlashPagedBlockAttentionKernel(Queue&, Tensor& out, const Tensor& query, }); } +// DFlash2 grouped dynamic depthwise convolution (SPEC-DFLASH2 W2, #1314) — the +// CPU REFERENCE, and the authoritative implementation the CUDA kernel mirrors. +// +// BEYOND-PIN. Ported from `_grouped_conv` +// (vllm/model_executor/models/qwen3_dflash2.py @ vllm-project/vllm#52816 head +// `19c9351904df4c63042671bc67a866ca48dc7d6f`): +// +// blocks = hidden_states.unflatten(-1, (num_groups, group_size)) +// coefficients = base.view(1, taps, num_groups, group_size) + delta.unsqueeze(-1) +// output = coefficients[:, 0] * blocks +// position = torch.arange(hidden_states.shape[0], device=...) +// if block_size & (block_size - 1) == 0: +// position = position & (block_size - 1) +// else: +// position = position % block_size +// for tap in range(1, taps): +// shifted = F.pad(blocks[:-tap], (0, 0, 0, 0, tap, 0)) +// output += coefficients[:, tap] * shifted * (position >= tap).view(-1, 1, 1) +// return output.flatten(-2) +// +// Three things are load-bearing and each is invisible to a token gate if wrong, +// because a DFlash2 draft with a broken conv still emits the TARGET's tokens +// (the verify is lossless) and loses only acceptance: +// +// 1. THE BLOCK MASK. `position` is the GLOBAL row index reduced modulo the +// block, so tap `t` contributes only where the source row `i-t` lies in the +// same (1+k) query block. Upstream's `F.pad(blocks[:-tap], ...)` supplies a +// zero for the first `tap` rows of the WHOLE batch and the mask supplies it +// at every later block boundary; this loop simply stops at `t > pos`, which +// is the same set because the mask is monotone in `t`. +// 2. THE GROUP MAP. `delta` is indexed per GROUP and `base` per CHANNEL, so the +// same delta applies to every channel of a group. `g = c / group_size`. +// 3. THE SIDE. `base` is `[sides, taps, H]` and dim 0 is prepare/finish, NOT a +// tap. On the published 27B draft both axes are 2, so a swap is undetectable +// by shape alone. +// +// ROUNDING. Upstream's chain materializes a tensor of the model dtype after each +// step (`base + delta`, `coefficients * blocks`, `output += ...`), so each step +// rounds here too. Every step is elementwise with no reduction-order freedom, so +// this and the CUDA kernel agree BIT-FOR-BIT rather than within an envelope. +void DFlashGroupedConvKernel(Queue&, Tensor& out, const Tensor& x, const Tensor& coefficients, + const Tensor& base, const DFlashGroupedConvArgs& args) { + const int64_t rows = x.shape[0]; + const int64_t taps = args.taps; + const int64_t groups = args.num_groups; + const int64_t gsize = args.group_size; + const int64_t h = groups * gsize; + const int64_t sides = coefficients.shape[1]; + const int64_t side = args.side; + const int64_t block = args.block_size; + // Upstream's own power-of-two special case, mirrored including the `&` arm: + // both published DFlash2 checkpoints resolve to a power-of-two query block + // (8 and 16), and upstream's reference test parametrises 5 for the other arm. + const bool pot = (block & (block - 1)) == 0; + const DType dt = out.dtype; + const auto round = [dt](float v) -> float { + switch (dt) { + case DType::kF32: return v; + case DType::kF16: return F16ToF32(F32ToF16(v)); + case DType::kBF16: return BF16ToF32(F32ToBF16(v)); + default: VT_CHECK(false, "dflash2-grouped-conv: unsupported dtype"); return 0.0f; + } + }; + ForRows(rows, [&](int64_t r0, int64_t r1) { + for (int64_t i = r0; i < r1; ++i) { + const int64_t pos = pot ? (i & (block - 1)) : (i % block); + for (int64_t g = 0; g < groups; ++g) { + for (int64_t j = 0; j < gsize; ++j) { + const int64_t c = g * gsize + j; + float acc = 0.0f; + for (int64_t t = 0; t < taps && t <= pos; ++t) { + const float b = LoadF32(base, (side * taps + t) * h + c); + const float d = LoadF32(coefficients, ((i * sides + side) * taps + t) * groups + g); + const float k = round(b + d); + const float term = round(k * LoadF32(x, (i - t) * h + c)); + acc = (t == 0) ? term : round(acc + term); + } + StoreF32(out, i * h + c, acc); + } + } + } + }); +} + // --- Qwen3.6 elementwise "glue" ops (M0.9 forward). Elementwise fusions of the // small host-side loops between the big decode ops; all math f32, dims inferred // from the tensor shapes. @@ -3450,6 +3534,9 @@ struct Registrar { RegisterOp(OpId::kDFlashPagedBlockAttention, DeviceType::kCPU, reinterpret_cast( static_cast(&DFlashPagedBlockAttentionKernel))); + RegisterOp(OpId::kDFlashGroupedConv, DeviceType::kCPU, + reinterpret_cast( + static_cast(&DFlashGroupedConvKernel))); RegisterOp(OpId::kCastBf16, DeviceType::kCPU, reinterpret_cast(static_cast(&CastBf16Kernel))); RegisterOp(OpId::kCastF32, DeviceType::kCPU, diff --git a/src/vt/cuda/cuda_ops.cu b/src/vt/cuda/cuda_ops.cu index 595c2694b..e72cd1059 100644 --- a/src/vt/cuda/cuda_ops.cu +++ b/src/vt/cuda/cuda_ops.cu @@ -3557,6 +3557,73 @@ void FusedChainKernelCuda(Queue& q, Tensor& out, const Tensor& x, const Tensor& } } +// --------------------------------------------------------------------------- +// DFlash2 grouped dynamic depthwise convolution (SPEC-DFLASH2 W2, #1314) — the +// CUDA MIRROR of the CPU reference `DFlashGroupedConvKernel` (cpu_ops.cpp), +// which carries the full port note and the upstream anchor. +// +// One thread per (row, channel). Every step is elementwise, so this kernel has +// NO reduction-order freedom and is asserted BIT-IDENTICAL to the CPU reference +// rather than within an envelope +// (tests/vt/test_ops_dflash2_grouped_conv.cpp). +// +// The two rounding helpers are `__fadd_rn`/`__fmul_rn` rather than `+`/`*` on +// purpose: on the f32 arm `ResRound` is the identity, so `acc + k * x` is an +// FMA contraction pattern and nvcc would fuse it by default, which the CPU +// reference (built with `-ffp-contract=off`) does not. The intrinsics forbid +// the contraction, so the two arms answer the same bits. +template +__global__ void DFlashGroupedConvKernel(T* out, const T* x, const T* coefficients, const T* base, + int64_t rows, int64_t h, int64_t taps, int64_t groups, + int64_t gsize, int64_t sides, int64_t side, + int64_t block, bool pot) { + const int64_t idx = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; + if (idx >= rows * h) return; + const int64_t i = idx / h; + const int64_t c = idx - i * h; + const int64_t g = c / gsize; + const int64_t pos = pot ? (i & (block - 1)) : (i % block); + float acc = 0.0f; + for (int64_t t = 0; t < taps && t <= pos; ++t) { + const float b = Load(base, (side * taps + t) * h + c); + const float d = Load(coefficients, ((i * sides + side) * taps + t) * groups + g); + const float k = ResRound(__fadd_rn(b, d)); + const float term = ResRound(__fmul_rn(k, Load(x, (i - t) * h + c))); + acc = (t == 0) ? term : ResRound(__fadd_rn(acc, term)); + } + Store(out, idx, acc); +} + +void DFlashGroupedConvKernelCuda(Queue& q, Tensor& out, const Tensor& x, + const Tensor& coefficients, const Tensor& base, + const DFlashGroupedConvArgs& args) { + const int64_t rows = x.shape[0]; + const int64_t h = args.num_groups * args.group_size; + const int64_t sides = coefficients.shape[1]; + const int64_t n = rows * h; + if (n == 0) return; + constexpr int kBlock = 256; + const int64_t grid = (n + kBlock - 1) / kBlock; + const bool pot = (args.block_size & (args.block_size - 1)) == 0; + cudaStream_t s = AsStream(q); + switch (x.dtype) { + case DType::kF32: + DFlashGroupedConvKernel<<(grid), kBlock, 0, s>>>( + out.Ptr(), x.Ptr(), coefficients.Ptr(), base.Ptr(), rows, h, + args.taps, args.num_groups, args.group_size, sides, args.side, args.block_size, pot); + break; + case DType::kBF16: + DFlashGroupedConvKernel<__nv_bfloat16><<(grid), kBlock, 0, s>>>( + out.Ptr<__nv_bfloat16>(), x.Ptr<__nv_bfloat16>(), coefficients.Ptr<__nv_bfloat16>(), + base.Ptr<__nv_bfloat16>(), rows, h, args.taps, args.num_groups, args.group_size, sides, + args.side, args.block_size, pot); + break; + default: + VT_CHECK(false, "cuda dflash2-grouped-conv: unsupported dtype (f32/bf16 only)"); + } + VT_CHECK(cudaGetLastError() == cudaSuccess, "cuda dflash2-grouped-conv: launch failed"); +} + // Registers the CUDA kernels during static init (pre-main, like the CPU ops). // Filling the op table is harmless on machines without a GPU: the kCUDA // backend never registers there, so no CUDA queue can exist to dispatch with. @@ -3608,6 +3675,9 @@ struct Registrar { static_cast(&DFlashPagedBlockAttentionKernelCuda))); RegisterOp(OpId::kFusedChain, DeviceType::kCUDA, reinterpret_cast(static_cast(&FusedChainKernelCuda))); + RegisterOp(OpId::kDFlashGroupedConv, DeviceType::kCUDA, + reinterpret_cast( + static_cast(&DFlashGroupedConvKernelCuda))); } } registrar; diff --git a/src/vt/op_provider.cpp b/src/vt/op_provider.cpp index 24f6432d7..1af430ef4 100644 --- a/src/vt/op_provider.cpp +++ b/src/vt/op_provider.cpp @@ -292,6 +292,8 @@ const char* OpNameImpl(OpId op) { return "DFlashBlockAttention"; case OpId::kDFlashPagedBlockAttention: return "DFlashPagedBlockAttention"; + case OpId::kDFlashGroupedConv: + return "DFlashGroupedConv"; case OpId::kReshapeAndCache: return "ReshapeAndCache"; case OpId::kConcatAndCacheMla: diff --git a/src/vt/ops.cpp b/src/vt/ops.cpp index 26b8d78d1..1aadc4327 100644 --- a/src/vt/ops.cpp +++ b/src/vt/ops.cpp @@ -3123,6 +3123,46 @@ void DFlashPagedBlockAttention(Queue& q, Tensor& out, const Tensor& query, args); } +void DFlashGroupedConv(Queue& q, Tensor& out, const Tensor& x, const Tensor& coefficients, + const Tensor& base, const DFlashGroupedConvArgs& args) { + VT_CHECK(args.taps >= 1 && args.num_groups >= 1 && args.group_size >= 1, + "dflash2-grouped-conv: taps/num_groups/group_size must be >= 1"); + VT_CHECK(args.block_size >= 1, "dflash2-grouped-conv: block_size must be >= 1 (1 + k)"); + VT_CHECK(x.rank == 2 && out.rank == 2, "dflash2-grouped-conv: x/out must be rank-2 [T,H]"); + VT_CHECK(coefficients.rank == 4, + "dflash2-grouped-conv: coefficients must be rank-4 [T,sides,taps,num_groups]"); + VT_CHECK(base.rank == 3, "dflash2-grouped-conv: base must be rank-3 [sides,taps,H]"); + const int64_t t = x.shape[0]; + const int64_t h = args.num_groups * args.group_size; + const int64_t sides = coefficients.shape[1]; + VT_CHECK(x.shape[1] == h, + "dflash2-grouped-conv: x hidden must be num_groups*group_size"); + VT_CHECK(out.shape[0] == t && out.shape[1] == h, + "dflash2-grouped-conv: out must be [T,H] matching x"); + VT_CHECK(coefficients.shape[0] == t && coefficients.shape[2] == args.taps && + coefficients.shape[3] == args.num_groups, + "dflash2-grouped-conv: coefficients must be [T,sides,taps,num_groups]"); + VT_CHECK(base.shape[0] == sides && base.shape[1] == args.taps && base.shape[2] == h, + "dflash2-grouped-conv: base must be [sides,taps,H] with the coefficients' sides"); + VT_CHECK(args.side >= 0 && args.side < sides, + "dflash2-grouped-conv: side must index the sides dimension " + "(0 = prepare, 1 = finish)"); + // ONE dtype across all four. The rounding after each step is what makes the + // CPU reference and the CUDA kernel bit-identical, and a mixed set would make + // "the dtype" ambiguous rather than merely inconvenient. + VT_CHECK(IsFloat(x.dtype) && coefficients.dtype == x.dtype && base.dtype == x.dtype && + out.dtype == x.dtype, + "dflash2-grouped-conv: x/coefficients/base/out must share one float dtype"); + VT_CHECK(x.IsContiguous() && coefficients.IsContiguous() && base.IsContiguous() && + out.IsContiguous(), + "dflash2-grouped-conv: contiguous tensors required"); + VT_CHECK(x.device == q.device && coefficients.device == q.device && + base.device == q.device && out.device == q.device, + "dflash2-grouped-conv: device mismatch (x/coefficients/base/out/queue)"); + reinterpret_cast(GetOp(OpId::kDFlashGroupedConv, q.device.type))( + q, out, x, coefficients, base, args); +} + void ReshapeAndCache(Queue& q, const Tensor& k, const Tensor& v, Tensor& k_cache, Tensor& v_cache, const Tensor& slot_mapping) { VT_CHECK(k.rank == 3 && v.rank == 3, diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index cbef7b53e..001e9f92f 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -552,6 +552,11 @@ vllm_cpp_add_test(test_qwen3_forward vllm/models/test_qwen3_forward.cpp) target_include_directories(test_qwen3_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_qwen3_dflash_forward vllm/models/test_qwen3_dflash_forward.cpp) target_include_directories(test_qwen3_dflash_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) +# SPEC-DFLASH2 W2 (#1314): the DFlash2 draft -- the published configs' nested +# spellings (spec `## Owed` O3/O4), the conv weights, and the draft forward that +# runs the grouped convolution. +vllm_cpp_add_test(test_qwen3_dflash2_draft vllm/models/test_qwen3_dflash2_draft.cpp) +target_include_directories(test_qwen3_dflash2_draft PRIVATE ${CMAKE_SOURCE_DIR}/src) # SPEC-DSPARK W2: the DSpark Markov transition head + draft->target vocab map. vllm_cpp_add_test(test_qwen3_dspark_markov vllm/models/test_qwen3_dspark_markov.cpp) target_include_directories(test_qwen3_dspark_markov PRIVATE ${CMAKE_SOURCE_DIR}/src) @@ -1305,6 +1310,10 @@ target_include_directories(test_dflash_kvprep PRIVATE ${CMAKE_SOURCE_DIR}/src) # assertion that a DFlash1 checkpoint's resolution is unchanged. vllm_cpp_add_test(test_dflash_causality vllm/v1/spec_decode/test_dflash_causality.cpp) vllm_cpp_add_test(test_dflash_propose vllm/v1/spec_decode/test_dflash_propose.cpp) +# SPEC-DFLASH2 W2 (#1314): the candidate-selector refusal -- the boundary the +# DFlash2 architecture is left at once its grouped convolution runs. +vllm_cpp_add_test(test_dflash2_selector_refusal + vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp) target_include_directories(test_dflash_propose PRIVATE ${CMAKE_SOURCE_DIR}/src) # SPEC-NGRAM (ROAD-V1-D3) — the draft-free n-gram matcher unit gate (ports # vllm/tests/v1/spec_decode/test_ngram.py, host-side, runs everywhere). @@ -2047,6 +2056,10 @@ vllm_cpp_add_test(test_ops_attention_dense_fa2 vt/test_ops_attention_dense_fa2.c vllm_cpp_add_test(test_ops_attention_cross vt/test_ops_attention_cross.cpp) vllm_cpp_add_test(test_ops_dflash_block_attn vt/test_ops_dflash_block_attn.cpp) vllm_cpp_add_test(test_ops_dflash_paged_block_attn vt/test_ops_dflash_paged_block_attn.cpp) +# SPEC-DFLASH2 W2 (#1314): the DFlash2 grouped dynamic depthwise convolution, +# gated against upstream's own sequential reference at block 5 (modulo arm), 8 +# and 16 (the two published checkpoints) plus CUDA==CPU bit-identity. +vllm_cpp_add_test(test_ops_dflash2_grouped_conv vt/test_ops_dflash2_grouped_conv.cpp) vllm_cpp_add_test(test_ops_reshape_cache vt/test_ops_reshape_cache.cpp) # KV-FP8 W1: fp8 KV-cache store (ReshapeAndCacheFp8) + paged-attention read # dequant + the ParseCacheDType config wiring. diff --git a/tests/vllm/entrypoints/test_dflash2_draft_routing.cpp b/tests/vllm/entrypoints/test_dflash2_draft_routing.cpp index 2353527ef..f9411f933 100644 --- a/tests/vllm/entrypoints/test_dflash2_draft_routing.cpp +++ b/tests/vllm/entrypoints/test_dflash2_draft_routing.cpp @@ -45,7 +45,9 @@ #include #include #include +#include #include +#include #include #include @@ -188,32 +190,49 @@ std::string RefusalForDraft(const std::string& draft_path) { } // namespace -TEST_CASE("the loader refuses a DFlash2 draft instead of drafting it as DFlash1") { +TEST_CASE("W2: a safetensors DFlash2 draft is now ADMITTED as far as the conv") { + // W1 REFUSED this draft here, before any weight was read, because BOTH + // mechanisms were missing. SPEC-DFLASH2 W2 implements one of them -- the + // grouped dynamic depthwise convolution -- and a startup refusal would leave + // every line of it unreachable from any production entry point, which is what + // AGENTS.md `## Nothing lands dead` forbids. So the draft is admitted here and + // refused one step later, at the candidate selector, AFTER the conv has run + // (`RefuseDflash2CandidateSelector`, gated in + // tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp). + // + // What must NOT happen is a fall-through into a resolved config with nothing + // named: the boundary is stated at STARTUP so the later refusal is not a + // surprise. That notice goes to stderr and is asserted below. const ScratchDraft draft(kDflash2DraftConfig); - const std::string message = RefusalForDraft(draft.path()); - REQUIRE_FALSE(message.empty()); // RED before this row: nothing classifies. - // AGENTS.md `## Shared seams`: refuse with a message that NAMES the missing - // part. Both mechanisms, because a user who reads only "DFlash2" cannot tell - // what is absent, and both are separately owed by the spec's `## Work - // breakdown` (W2 and W3). - INFO("what: ", message); - CHECK(message.find("DFlash2DraftModel") != std::string::npos); - CHECK(message.find("grouped dynamic") != std::string::npos); - CHECK(message.find("candidate selector") != std::string::npos); - CHECK(message.find("not implemented") != std::string::npos); - // The row and the issue, so the reader can find who owns the wiring. - CHECK(message.find("SPEC-DFLASH2") != std::string::npos); - CHECK(message.find("#1314") != std::string::npos); + CHECK(RefusalForDraft(draft.path()).empty()); + const std::optional cfg = + LoadedEngine::ResolveSpecConfig(DflashParams(draft.path(), 8), vllm::HfConfig{}); + REQUIRE(cfg.has_value()); + CHECK(cfg->method == "dflash"); + CHECK(cfg->ResolvedNumSpeculativeTokens() == 8); } -TEST_CASE("the DFlash2 refusal THROWS rather than resolving a dflash config") { - // The failure mode this row removes is not a wrong message, it is a resolved - // config: falling through leaves a spec-ON engine whose draft silently lacks - // the convolution and the selector. +TEST_CASE("W2: admitting the DFlash2 draft STATES the boundary at startup") { + // A user who is admitted silently and refused at the first generated token has + // been told nothing. The notice names the mechanism that runs, the one that + // does not, the wave that owns it and the issue. const ScratchDraft draft(kDflash2DraftConfig); - CHECK_THROWS_AS( - LoadedEngine::ResolveSpecConfig(DflashParams(draft.path(), 8), vllm::HfConfig{}), - std::invalid_argument); + std::ostringstream captured; + std::streambuf* const previous = std::cerr.rdbuf(captured.rdbuf()); + try { + (void)LoadedEngine::ResolveSpecConfig(DflashParams(draft.path(), 8), vllm::HfConfig{}); + } catch (...) { + std::cerr.rdbuf(previous); + throw; + } + std::cerr.rdbuf(previous); + const std::string notice = captured.str(); + INFO("notice: ", notice); + CHECK(notice.find("DFlash2DraftModel") != std::string::npos); + CHECK(notice.find("grouped dynamic") != std::string::npos); + CHECK(notice.find("CANDIDATE SELECTOR") != std::string::npos); + CHECK(notice.find("SPEC-DFLASH2") != std::string::npos); + CHECK(notice.find("#1314") != std::string::npos); } TEST_CASE("the loader still admits a DFlashDraftModel draft") { @@ -239,7 +258,29 @@ TEST_CASE("a draft with no config.json to read resolves exactly as before") { CHECK(RefusalForDraft(draft.path()).empty()); } -TEST_CASE("the loader refuses a DFlash2 draft BEFORE it touches the model directory") { +TEST_CASE("W2: the early FromModelDir guard no longer refuses a safetensors DFlash2 draft") { + // The mirror of the case above at the OTHER production call site. `FromModelDir` + // loads the dflash draft before it builds the `LoadedEngine`, so W1 guarded it + // separately; W2 admits the safetensors arm at both sites, and the failure a + // user now sees for a nonexistent target is the target error, not a DFlash2 + // refusal. The GGUF arm below still refuses at this same site. + const ScratchDraft draft(kDflash2DraftConfig); + EngineParams params = DflashParams(draft.path(), 8); + std::ostringstream sink; + std::streambuf* const previous = std::cerr.rdbuf(sink.rdbuf()); + std::string what; + try { + (void)LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/dflash2/target", params); + } catch (const std::exception& e) { + what = e.what(); + } + std::cerr.rdbuf(previous); + INFO("what: ", what); + CHECK(what.find("model path is not a directory") != std::string::npos); + CHECK(what.find("not implemented") == std::string::npos); +} + +TEST_CASE("the loader refuses a DFlash2 GGUF draft BEFORE it touches the model directory") { // The SECOND production call site, and the one the constructor's resolution // cannot cover. `FromModelDir` loads a dflash draft (`maybe_load_dflash`) // BEFORE it builds the `LoadedEngine`, and that site resolves the draft from @@ -251,15 +292,15 @@ TEST_CASE("the loader refuses a DFlash2 draft BEFORE it touches the model direct // directory, so the refusal is proven to land ahead of every path, config, // tokenizer and weight operation. RED without the guard: the failure is // "model path is not a directory" and the draft classification never runs. - const ScratchDraft draft(kDflash2DraftConfig); + const gguf_test::TempFile draft(DflashGgufBytes(/*dflash2=*/true)); EngineParams params = DflashParams(draft.path(), 8); try { (void)LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/dflash2/target", params); - FAIL("expected a refusal for a DFlash2 draft"); + FAIL("expected a refusal for a DFlash2 GGUF draft"); } catch (const std::exception& e) { const std::string what = e.what(); INFO("what: ", what); - CHECK(what.find("DFlash2DraftModel") != std::string::npos); + CHECK(what.find("GGUF drafter ARM") != std::string::npos); CHECK(what.find("model path is not a directory") == std::string::npos); } } diff --git a/tests/vllm/models/test_qwen3_dflash2_draft.cpp b/tests/vllm/models/test_qwen3_dflash2_draft.cpp new file mode 100644 index 000000000..ea2bf6b1a --- /dev/null +++ b/tests/vllm/models/test_qwen3_dflash2_draft.cpp @@ -0,0 +1,942 @@ +// SPEC-DFLASH2 W2 (#1314) — the DFlash2 draft: its config, its grouped dynamic +// convolution weights, and the forward that runs them. +// +// BEYOND-PIN throughout. Upstream is +// `vllm/model_executor/models/qwen3_dflash2.py` @ vllm-project/vllm#52816 head +// `19c9351904df4c63042671bc67a866ca48dc7d6f`; the parity pin `555967922` does +// not carry the architecture and this row does not advance it. +// +// PART 1 — the config builder, which is `## Owed` O3 and O4 of the row's spec. +// `MakeQwen3DFlashDraftConfig` could not parse EITHER published DFlash2 +// `config.json` at all: it did `c.at("rope_theta")` and `c.at("block_size")` +// while `z-lab/Qwen3.8-27B-DFlash2` nests them as `rope_parameters.rope_theta` +// and `dflash_config.block_size` and declares neither at the top level, so both +// `at` calls threw before any DFlash2 mechanism could be reached. It also did +// `c.at("layer_types")`, which `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` does not +// declare, while upstream reads `getattr(config, "layer_types", None)` +// (`qwen3_dflash.py:134` and `:66` @ that head). The configs embedded below are +// the PUBLISHED files verbatim, with their sha256 recorded, so the gate does not +// depend on a checkout being present. +// +// The `attention_sink_bias` refusal is the other half of O4 and is not +// bookkeeping. Upstream reads `dflash_config.attention_sink_bias` and passes a +// per-head sink into its `Attention` (`qwen3_dflash.py:309-313` and `:240-257` @ +// that head); this lane has no attention sink at all. Landing the `layer_types` +// fallback ALONE would let MiMo's draft parse and load with the sinks silently +// absent -- acceptance-only, token-invisible, which is the exact class this row +// exists to remove. So the key is refused BY NAME. +// +// PART 2 — the grouped convolution inside the draft forward. The op itself is +// gated in tests/vt/test_ops_dflash2_grouped_conv.cpp against upstream's own +// sequential reference; what is gated here is that the DRAFT MODEL runs it, on +// weights the PRODUCTION loader read off a real on-disk checkpoint, and that +// deleting the production call site turns this suite red. +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/qwen3_dflash.h" +#include "vllm/v1/worker/gpu/spec_decode/dflash/speculator.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace fs = std::filesystem; +using nlohmann::json; +using vllm::HfConfig; +using vllm::MakeQwen3DFlashDraftConfig; +using vllm::OwnedTensor; +using vllm::Qwen3DFlashLayerAttnMode; +using vllm::Qwen3DFlashModel; +using vllm::Qwen3DFlashWeights; +using vllm::ResolveQwen3DFlashAttnModes; + +namespace { + +// `z-lab/Qwen3.8-27B-DFlash2` @ `50307d4c4cde6860d4eee73e2547cd786fe8e8a4`, +// config.json VERBATIM (1239 bytes, sha256 +// 873e3556509b0da06e29654ba00d4944888d4b5e8a33afde25f7eb27d321e980, read on +// 2026-08-19). Kept whole rather than reduced: what this case gates is that the +// PUBLISHED document parses, and a reduced copy would only prove that a document +// this test wrote parses. +constexpr const char* kQwen38Dflash2Config = R"JSON({ + "architectures": [ + "DFlash2DraftModel" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "is_causal": false, + "dflash_config": { + "block_size": 8, + "conv_group_size": 16, + "conv_kernel_size": 2, + "mask_token_id": 248070, + "selector_rank": 256, + "selector_top_k": 16, + "target_layer_ids": [ + 5, + 19, + 33, + 47, + 61 + ] + }, + "dtype": "bfloat16", + "eos_token_id": 248044, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 5120, + "initializer_range": 0.02, + "intermediate_size": 17408, + "layer_types": [ + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention" + ], + "max_position_embeddings": 262144, + "max_window_layers": 5, + "model_type": "qwen3", + "num_attention_heads": 32, + "num_hidden_layers": 5, + "num_key_value_heads": 8, + "num_target_layers": 64, + "pad_token_id": 248044, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 10000000, + "rope_type": "default" + }, + "sliding_window": 2048, + "tie_word_embeddings": false, + "transformers_version": "5.15.0", + "use_cache": true, + "use_sliding_window": true, + "vocab_size": 248320 +})JSON"; + +// `z-lab/Muse-Glimmer-30B-DFlash2`, config.json VERBATIM (sha256 +// cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9, read on +// 2026-08-19). The SECOND published DFlash2 checkpoint, and the one that makes +// #1327 a correction rather than a note: `block_size` 16 against the 27B's 8, +// and `output_multiplier`/`final_logit_softcapping` SET rather than defaulted. +constexpr const char* kMuseGlimmerDflash2Config = R"JSON({ + "architectures": [ + "DFlash2DraftModel" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 200000, + "is_causal": false, + "dflash_config": { + "block_size": 16, + "conv_group_size": 16, + "conv_kernel_size": 2, + "final_logit_softcapping": 20.0, + "mask_token_id": 201818, + "output_multiplier": 0.19611613513818404, + "selector_rank": 256, + "selector_top_k": 16, + "target_layer_ids": [ + 1, + 13, + 25, + 37, + 49 + ] + }, + "dtype": "bfloat16", + "eos_token_id": 200001, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 6656, + "initializer_range": 0.02, + "intermediate_size": 19968, + "layer_types": [ + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention" + ], + "max_position_embeddings": 131072, + "max_window_layers": 5, + "model_type": "qwen3", + "num_attention_heads": 32, + "num_hidden_layers": 5, + "num_key_value_heads": 8, + "num_target_layers": 52, + "pad_token_id": 200018, + "rms_norm_eps": 1e-05, + "rope_parameters": { + "rope_theta": 500000.0, + "rope_type": "default" + }, + "sliding_window": 2048, + "tie_word_embeddings": false, + "transformers_version": "5.15.0", + "use_cache": false, + "use_sliding_window": true, + "vocab_size": 202048 +})JSON"; + +// `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` `dflash/config.json` VERBATIM (sha256 +// 2ed5a998f5f57e00a9fe14d2b3e767f06e49462a97eb09d80c927e112a585c9e, read on +// 2026-08-19). A DFlash1 draft, present here for O4: it is the ONLY published +// draft that declares no `layer_types`, and it is also the only one that +// declares `attention_sink_bias`. +constexpr const char* kMimoDflashConfig = R"JSON({ + "architectures": ["DFlashDraftModel"], + "model_type": "qwen3", + "hidden_size": 6144, + "intermediate_size": 16384, + "num_hidden_layers": 5, + "num_attention_heads": 128, + "num_key_value_heads": 8, + "head_dim": 128, + "v_head_dim": 128, + "partial_rotary_factor": 0.5, + "block_size": 8, + "dflash_config": { + "target_layer_ids": [0, 15, 31, 47, 69], + "mask_token_id": 151669, + "num_anchors": 4096, + "block_size": 8, + "loss_decay_gamma": 7.0, + "use_swa": true, + "swa_window_size": 1024, + "backbone_rotary_base": 5000000, + "attention_value_scale": 0.612, + "attention_sink_bias": true + }, + "num_target_layers": 70, + "vocab_size": 152064, + "max_position_embeddings": 262144, + "rope_theta": 10000, + "sliding_window": 1024, + "rms_norm_eps": 1e-05, + "torch_dtype": "bfloat16", + "hidden_act": "silu", + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 151643, + "eos_token_id": 151645, + "tie_word_embeddings": false, + "use_cache": true +})JSON"; + +} // namespace + +TEST_CASE("dflash2 config: the published Qwen3.8-27B DFlash2 config.json PARSES (O3)") { + const HfConfig c = MakeQwen3DFlashDraftConfig(json::parse(kQwen38Dflash2Config)); + CHECK(c.hidden_size == 5120); + CHECK(c.num_hidden_layers == 5); + CHECK(c.vocab_size == 248320); + // The nested spellings, which the flat reads could not see. `transformers` + // moved RoPE settings under `rope_parameters`, so this is a FALLBACK and not a + // replacement: DFlash1 checkpoints still carry the flat `rope_theta`. + CHECK(c.rope_theta == doctest::Approx(1e7)); + REQUIRE(c.raw.contains("block_size")); + CHECK(c.raw.at("block_size").get() == 8); + // The DFlash2 conv geometry the W2 forward needs, carried through dflash_config. + REQUIRE(c.raw.contains("dflash_config")); + CHECK(c.raw.at("dflash_config").at("conv_kernel_size").get() == 2); + CHECK(c.raw.at("dflash_config").at("conv_group_size").get() == 16); + // W1's rule, now reachable for the first time on a checkpoint that declares it. + REQUIRE(c.raw.contains("is_causal")); + const std::vector modes = ResolveQwen3DFlashAttnModes(c); + REQUIRE(modes.size() == 5); + for (size_t i = 0; i < modes.size(); ++i) { + CAPTURE(i); + CHECK_FALSE(modes[i].causal); + CHECK(modes[i].sliding_window == 2048); + } +} + +TEST_CASE("dflash2 config: the published Muse-Glimmer-30B DFlash2 config.json PARSES (#1327)") { + const HfConfig c = MakeQwen3DFlashDraftConfig(json::parse(kMuseGlimmerDflash2Config)); + CHECK(c.hidden_size == 6656); + CHECK(c.rope_theta == doctest::Approx(5e5)); + REQUIRE(c.raw.contains("block_size")); + CHECK(c.raw.at("block_size").get() == 16); + // hidden 6656 / conv_group_size 16 = 416 groups, so kernel_projection is + // 2*taps*num_groups = 1664 wide. The 27B's is 1280. The two shapes are what + // makes "gate at BOTH published blocks" a real requirement rather than a + // parameter sweep. + CHECK(c.raw.at("dflash_config").at("conv_group_size").get() == 16); + CHECK(c.hidden_size % c.raw.at("dflash_config").at("conv_group_size").get() == 0); + // The two scalars #1327 corrects the spec about: they are CHECKPOINT-exercised, + // not synthetic, and both are applied to candidate VALUES before the selector + // scores them, so a wrong one reorders the top-K and moves acceptance without + // raising. W3 consumes them; W2 asserts they survive the parse. + CHECK(c.raw.at("dflash_config").at("output_multiplier").get() == + doctest::Approx(0.19611613513818404)); + CHECK(c.raw.at("dflash_config").at("final_logit_softcapping").get() == + doctest::Approx(20.0)); +} + +TEST_CASE("dflash draft config: an ABSENT layer_types is upstream's None (O4)") { + // `getattr(config, "layer_types", None)` (qwen3_dflash.py:134 and :66 @ the PR + // head). As shipped this threw `[json.exception.out_of_range.403] key + // 'layer_types' not found` before any causality could be resolved, which is why + // #1366's `use_swa` repair was UNREACHED at its own merge commit. + json doc = json::parse(kMimoDflashConfig); + doc["dflash_config"].erase("attention_sink_bias"); // refused separately, below + const HfConfig c = MakeQwen3DFlashDraftConfig(doc); + CHECK(c.layer_types.empty()); + // Upstream's own `_resolve_layer_attention` docstring row: `layer_types=None` + // + `use_swa=True` -> SWA window, causal FALSE. Before #1366 this engine + // answered causal TRUE on every layer. + const std::vector modes = ResolveQwen3DFlashAttnModes(c); + REQUIRE(modes.size() == 5); + for (size_t i = 0; i < modes.size(); ++i) { + CAPTURE(i); + CHECK_FALSE(modes[i].causal); + CHECK(modes[i].sliding_window == 1024); + } +} + +TEST_CASE("dflash draft config: attention_sink_bias is REFUSED BY NAME (O4)") { + // Upstream reads it and passes a per-head sink into its Attention; this lane + // has none. Parsing the config WITHOUT refusing would convert a loud + // `key 'layer_types' not found` into a draft that loads with the sinks + // silently absent -- acceptance-only and invisible to a token gate. + const json doc = json::parse(kMimoDflashConfig); + CHECK_THROWS_WITH_AS(MakeQwen3DFlashDraftConfig(doc), + doctest::Contains("attention_sink_bias"), std::exception); + // A FALSE value is upstream's default and is not refused: `dflash_config.get( + // "attention_sink_bias", ...)` falsy means no sink parameter is created at all. + json off = doc; + off["dflash_config"]["attention_sink_bias"] = false; + CHECK_NOTHROW(MakeQwen3DFlashDraftConfig(off)); +} + +TEST_CASE("dflash draft config: the DFlash1 flat spellings are UNCHANGED") { + // The fallbacks must not move a published DFlash1 draft. `z-lab/Qwen3.6-27B-DFlash` + // declares a flat `rope_theta`, a flat `block_size` and a full `layer_types`, + // and every one of them must still win. + json doc = json::object(); + doc["hidden_size"] = 5120; + doc["num_attention_heads"] = 32; + doc["num_key_value_heads"] = 8; + doc["head_dim"] = 128; + doc["rope_theta"] = 1e7; + doc["intermediate_size"] = 25600; + doc["vocab_size"] = 248320; + doc["num_hidden_layers"] = 2; + doc["rms_norm_eps"] = 1e-6; + doc["sliding_window"] = 2048; + doc["layer_types"] = json::array({"sliding_attention", "full_attention"}); + doc["block_size"] = 17; + doc["dflash_config"] = json::object(); + doc["dflash_config"]["mask_token_id"] = 248070; + doc["dflash_config"]["target_layer_ids"] = json::array({5, 19}); + // A nested rope_parameters must NOT override a declared flat rope_theta on a + // document that carries both: upstream reads one resolved `config.rope_theta`. + const HfConfig c = MakeQwen3DFlashDraftConfig(doc); + CHECK(c.rope_theta == doctest::Approx(1e7)); + CHECK(c.raw.at("block_size").get() == 17); + REQUIRE(c.layer_types.size() == 2); + CHECK(c.layer_types[0] == "sliding_attention"); + CHECK(c.layer_types[1] == "full_attention"); + const std::vector modes = ResolveQwen3DFlashAttnModes(c); + REQUIRE(modes.size() == 2); + CHECK(modes[0].causal); + CHECK_FALSE(modes[1].causal); +} + +// =========================================================================== +// PART 2 — the conv INSIDE the draft, loaded by the production weight loader +// off a real on-disk safetensors checkpoint and run by the production forward. +// +// REACHABILITY (.agents/reachability.md). The chain a user arrives through is +// LoadedEngine::FromModelDir / ResolveSpecConfig +// -> LoadDflashDraft (src/vllm/entrypoints/model_loader.cpp) +// -> vllm::LoadQwen3DFlash (reads attention_conv/mlp_conv, sets +// conv_taps/conv_group_size; the loader then +// overwrites conv_block_size with 1 + k) +// -> GPUModelRunner::propose_drafts_block +// -> Qwen3DFlashModel::ForwardBlockLogitsWithDeviceKV +// -> the layer body -> DflashConvPrepare / DflashConvFinish +// -> vt::DFlashGroupedConv +// -> RefuseDflash2CandidateSelector (W3's boundary, by name) +// +// The cases below enter at `vllm::LoadQwen3DFlash` over a real safetensors file +// -- the same function the loader calls, on the same tensor names the published +// checkpoint uses -- and then at the model forward, which is what +// `propose_drafts_block` calls. The reachability MUTATION deletes the +// `DflashConvPrepare`/`DflashConvFinish` call sites in the layer body and this +// suite must redden. +// +// WHY AN IDENTITY CONV IS THE RIGHT PROBE. With `taps = 1`, `base_kernel` all +// ones and `kernel_projection` all zeros, the conv is EXACTLY the identity in +// bf16: `bf16(1 + 0) = 1` and `bf16(1 * x) = x`, with no tap to mask. So a +// DFlash2 draft carrying that conv must produce BIT-IDENTICAL logits to the same +// draft with no conv at all. That separates "the conv is wired into the right +// places and perturbs nothing it should not" from "the conv is wired somewhere", +// which a tolerance-based comparison cannot. + +namespace { + +// Minimal safetensors writer: header length (u64 LE) + header JSON + payload, +// which is the whole format. Mirrors the one in +// tests/vllm/multimodal/ltx2_video_fixture.h rather than adding a dependency on +// it, because that header carries an entire LTX-2.5 fixture with it. +struct StEntry { + std::string name; + std::vector shape; + std::vector bf16; +}; + +void WriteSafetensors(const std::vector& entries, const std::string& path) { + json header = json::object(); + size_t offset = 0; + for (const StEntry& e : entries) { + const size_t nbytes = e.bf16.size() * sizeof(uint16_t); + header[e.name] = {{"dtype", "BF16"}, + {"shape", e.shape}, + {"data_offsets", json::array({offset, offset + nbytes})}}; + offset += nbytes; + } + const std::string hs = header.dump(); + std::ofstream out(path, std::ios::binary); + uint64_t hlen = hs.size(); + out.write(reinterpret_cast(&hlen), sizeof(hlen)); + out.write(hs.data(), static_cast(hs.size())); + for (const StEntry& e : entries) + out.write(reinterpret_cast(e.bf16.data()), + static_cast(e.bf16.size() * sizeof(uint16_t))); +} + +// Deterministic bf16 fill: value(i) = amp * sin(seed + 0.7*i), the same shape of +// generator tests/vllm/models/test_qwen3_dflash_forward.cpp uses. +std::vector Fill(int64_t n, double seed, double amp) { + std::vector v(static_cast(n)); + for (int64_t i = 0; i < n; ++i) + v[static_cast(i)] = + vt::F32ToBF16(static_cast(amp * std::sin(seed + 0.7 * static_cast(i)))); + return v; +} + +std::vector Const(int64_t n, float value) { + return std::vector(static_cast(n), vt::F32ToBF16(value)); +} + +struct Dims { + int64_t H = 8, Hq = 2, Hkv = 1, Dh = 4, I = 6, vocab = 8, layers = 2, taps_fc = 2; + int64_t conv_taps = 0; // 0 = a DFlash1 checkpoint (no conv tensors at all) + int64_t conv_group = 4; + int64_t block = 8; // the conv's query block, 1 + k + // Which conv is non-identity, and on which side. An identity conv is + // taps=1/base=1/projection=0; a non-identity one gets a real projection. + bool attn_conv_active = false; + bool mlp_conv_active = false; + int active_side = -1; // -1 = both sides active; 0 = prepare only; 1 = finish only + // `base_kernel[side]`, per side. Both 1.0 with a zero projection is the exact + // bf16 identity; making the two DIFFER is what separates "the model passes + // side 0 to prepare and side 1 to finish" from "the model passes a side". + float base_side0 = 1.0f; + float base_side1 = 1.0f; +}; + +// A scratch directory holding one model.safetensors, removed on scope exit. +class ScratchCkpt { + public: + explicit ScratchCkpt(const std::vector& entries) { + static int counter = 0; + dir_ = fs::temp_directory_path() / + ("vllmcpp_dflash2_ckpt_" + std::to_string(counter++) + "_" + + std::to_string(static_cast(::getpid()))); + fs::create_directories(dir_); + WriteSafetensors(entries, (dir_ / "model.safetensors").string()); + } + ~ScratchCkpt() { + std::error_code ec; + fs::remove_all(dir_, ec); + } + ScratchCkpt(const ScratchCkpt&) = delete; + ScratchCkpt& operator=(const ScratchCkpt&) = delete; + std::string shard() const { return (dir_ / "model.safetensors").string(); } + + private: + fs::path dir_; +}; + +std::vector DraftEntries(const Dims& dm) { + const int64_t qdim = dm.Hq * dm.Dh, kdim = dm.Hkv * dm.Dh; + std::vector e; + e.push_back({"embed_tokens.weight", {dm.vocab, dm.H}, Fill(dm.vocab * dm.H, 0.1, 0.3)}); + e.push_back({"fc.weight", {dm.H, dm.H * dm.taps_fc}, Fill(dm.H * dm.H * dm.taps_fc, 0.2, 0.2)}); + e.push_back({"hidden_norm.weight", {dm.H}, Fill(dm.H, 0.3, 0.5)}); + e.push_back({"norm.weight", {dm.H}, Fill(dm.H, 0.4, 0.5)}); + e.push_back({"lm_head.weight", {dm.vocab, dm.H}, Fill(dm.vocab * dm.H, 0.5, 0.3)}); + for (int64_t l = 0; l < dm.layers; ++l) { + const std::string b = "layers." + std::to_string(l) + "."; + const double s = 1.0 + static_cast(l); + e.push_back({b + "input_layernorm.weight", {dm.H}, Fill(dm.H, s + 0.1, 0.6)}); + e.push_back({b + "post_attention_layernorm.weight", {dm.H}, Fill(dm.H, s + 0.2, 0.6)}); + e.push_back({b + "self_attn.q_proj.weight", {qdim, dm.H}, Fill(qdim * dm.H, s + 0.3, 0.25)}); + e.push_back({b + "self_attn.k_proj.weight", {kdim, dm.H}, Fill(kdim * dm.H, s + 0.4, 0.25)}); + e.push_back({b + "self_attn.v_proj.weight", {kdim, dm.H}, Fill(kdim * dm.H, s + 0.5, 0.25)}); + e.push_back({b + "self_attn.o_proj.weight", {dm.H, qdim}, Fill(dm.H * qdim, s + 0.6, 0.25)}); + e.push_back({b + "self_attn.q_norm.weight", {dm.Dh}, Fill(dm.Dh, s + 0.7, 0.7)}); + e.push_back({b + "self_attn.k_norm.weight", {dm.Dh}, Fill(dm.Dh, s + 0.8, 0.7)}); + e.push_back({b + "mlp.gate_proj.weight", {dm.I, dm.H}, Fill(dm.I * dm.H, s + 0.9, 0.25)}); + e.push_back({b + "mlp.up_proj.weight", {dm.I, dm.H}, Fill(dm.I * dm.H, s + 1.1, 0.25)}); + e.push_back({b + "mlp.down_proj.weight", {dm.H, dm.I}, Fill(dm.H * dm.I, s + 1.2, 0.25)}); + if (dm.conv_taps > 0) { + const int64_t groups = dm.H / dm.conv_group; + const int64_t proj_out = 2 * dm.conv_taps * groups; + for (int which = 0; which < 2; ++which) { + const std::string cp = b + (which == 0 ? "attention_conv." : "mlp_conv."); + const bool active = which == 0 ? dm.attn_conv_active : dm.mlp_conv_active; + // base_kernel is [2 SIDES, taps, H] and is ALL ONES: with a zero + // projection that is the exact bf16 identity at taps=1, and with a real + // projection it is upstream's `base + delta` with base 1. + // base_kernel[side][tap][channel]. Tap 0 carries the side's scale; every + // HIGHER tap is 1 when this conv is active and ZERO when it is not. + // + // The zero matters, and getting it wrong once already produced a gate + // that could not tell one missing call site from none: with base 1 on + // every tap and a zero projection, a taps=2 conv is `x[i] + x[i-1]` and + // NOT the identity, so an "inactive" conv still moved the logits and an + // "attention_conv only" arm was quietly exercising both convs. + std::vector base; + for (int side = 0; side < 2; ++side) { + const float scale = side == 0 ? dm.base_side0 : dm.base_side1; + for (int64_t t = 0; t < dm.conv_taps; ++t) { + const std::vector row = + Const(dm.H, t == 0 ? scale : (active ? 1.0f : 0.0f)); + base.insert(base.end(), row.begin(), row.end()); + } + } + e.push_back({cp + "base_kernel", {2, dm.conv_taps, dm.H}, base}); + std::vector proj(static_cast(proj_out * dm.H), vt::F32ToBF16(0.0f)); + if (active) { + const std::vector live = Fill(proj_out * dm.H, s + 2.0 + which, 0.4); + for (int64_t r = 0; r < proj_out; ++r) { + // Row r of the projection produces coefficient + // [side = r / (taps*groups)][tap][group]. Zeroing the rows of the + // side this case does not exercise is what makes "prepare only" and + // "finish only" separable. + const int64_t side = r / (dm.conv_taps * groups); + if (dm.active_side >= 0 && side != dm.active_side) continue; + for (int64_t c = 0; c < dm.H; ++c) + proj[static_cast(r * dm.H + c)] = live[static_cast(r * dm.H + c)]; + } + } + e.push_back({cp + "kernel_projection.weight", {proj_out, dm.H}, proj}); + } + } + } + return e; +} + +HfConfig DraftConfig(const Dims& dm) { + HfConfig c; + c.hidden_size = dm.H; + c.num_attention_heads = dm.Hq; + c.num_key_value_heads = dm.Hkv; + c.head_dim = dm.Dh; + c.rotary_dim = dm.Dh; + c.rope_theta = 1e7; + c.intermediate_size = dm.I; + c.vocab_size = dm.vocab; + c.num_hidden_layers = dm.layers; + c.rms_norm_eps = 1e-6; + c.sliding_window = 2048; + c.layer_types = std::vector(static_cast(dm.layers), "sliding_attention"); + c.raw = nlohmann::json::object(); + c.raw["dflash_config"] = json::object(); + c.raw["dflash_config"]["mask_token_id"] = 7; + if (dm.conv_taps > 0) { + c.raw["dflash_config"]["conv_kernel_size"] = dm.conv_taps; + c.raw["dflash_config"]["conv_group_size"] = dm.conv_group; + c.raw["dflash_config"]["block_size"] = dm.block; + } + c.raw["block_size"] = dm.block; + c.raw["is_causal"] = false; + return c; +} + +// Load through the PRODUCTION weight loader, off a real safetensors file. +Qwen3DFlashWeights LoadDraft(const ScratchCkpt& ck, const Dims& dm, const HfConfig& c) { + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(ck.shard())); + Qwen3DFlashWeights w = vllm::LoadQwen3DFlash(shards, c, dm.taps_fc, /*mask_token_id=*/7); + // What the loader does with the resolved k (src/vllm/entrypoints/model_loader.cpp, + // LoadDflashDraft): the conv's block is 1 + k and not the checkpoint key. + if (w.IsDflash2()) w.conv_block_size = dm.block; + return w; +} + +// One draft forward over a single uniform (1+k) query block, through the +// context-free body. `T` is the block, so `cu` is {0, T}. +std::vector Forward(const Qwen3DFlashWeights& w, const HfConfig& c, int64_t T) { + std::vector ids(static_cast(T)); + std::vector pos(static_cast(T)); + for (int64_t i = 0; i < T; ++i) { + ids[static_cast(i)] = static_cast(i % c.vocab_size); + pos[static_cast(i)] = static_cast(i); + } + const std::vector cu = {0, static_cast(T)}; + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + return Qwen3DFlashModel::ForwardBlockLogits(ids, pos, cu, w, c, q); +} + +// The same block through the CONTEXT-AWARE body -- the one +// `DflashProposeBlock` and the runner's gathered path call -- with an empty +// context, so the two bodies are exercised on identical inputs. +std::vector ForwardWithContext(const Qwen3DFlashWeights& w, const HfConfig& c, int64_t T) { + std::vector ids(static_cast(T)); + std::vector pos(static_cast(T)); + for (int64_t i = 0; i < T; ++i) { + ids[static_cast(i)] = static_cast(i % c.vocab_size); + pos[static_cast(i)] = static_cast(i); + } + const std::vector cu = {0, static_cast(T)}; + const std::vector ctx_cu = {0, 0}; + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + return Qwen3DFlashModel::ForwardBlockLogitsWithContext({}, {}, ctx_cu, ids, pos, cu, w, c, q); +} + +bool BitEqual(const std::vector& a, const std::vector& b) { + if (a.size() != b.size()) return false; + return std::memcmp(a.data(), b.data(), a.size() * sizeof(float)) == 0; +} + +} // namespace + +TEST_CASE("dflash2 weights: the production loader reads the conv tensors off a real shard") { + Dims dm; + dm.conv_taps = 2; + const ScratchCkpt ck(DraftEntries(dm)); + const HfConfig c = DraftConfig(dm); + const Qwen3DFlashWeights w = LoadDraft(ck, dm, c); + CHECK(w.IsDflash2()); + CHECK(w.conv_taps == 2); + CHECK(w.conv_group_size == 4); + REQUIRE(w.layers.size() == 2); + for (size_t l = 0; l < w.layers.size(); ++l) { + CAPTURE(l); + CHECK_FALSE(w.layers[l].attention_conv.Empty()); + CHECK_FALSE(w.layers[l].mlp_conv.Empty()); + // [2 SIDES, taps, H] -- dim 0 is prepare/finish, NOT a tap. + CHECK(w.layers[l].attention_conv.base_kernel.rank == 3); + CHECK(w.layers[l].attention_conv.base_kernel.shape[0] == 2); + CHECK(w.layers[l].attention_conv.base_kernel.shape[1] == dm.conv_taps); + CHECK(w.layers[l].attention_conv.base_kernel.shape[2] == dm.H); + // 2 * taps * num_groups = 2*2*(8/4) = 8, the shape upstream's + // kernel_projection produces (1280 at the published 27B's 5120/16). + CHECK(w.layers[l].mlp_conv.kernel_projection.shape[0] == 2 * dm.conv_taps * (dm.H / dm.conv_group)); + CHECK(w.layers[l].mlp_conv.kernel_projection.shape[1] == dm.H); + } +} + +TEST_CASE("dflash1 weights: a draft with no conv keys stays a DFlash1 draft") { + // The inertness half. A DFlash1 checkpoint declares no conv key and ships no + // conv tensor, so `IsDflash2()` is false and every layer body runs the + // pre-W2 op sequence -- which the identity case below turns into a + // BIT-IDENTICAL assertion rather than an argument. + Dims dm; // conv_taps 0 + const ScratchCkpt ck(DraftEntries(dm)); + const HfConfig c = DraftConfig(dm); + const Qwen3DFlashWeights w = LoadDraft(ck, dm, c); + CHECK_FALSE(w.IsDflash2()); + CHECK(w.conv_taps == 0); + CHECK(w.layers[0].attention_conv.Empty()); + CHECK(w.layers[0].mlp_conv.Empty()); +} + +TEST_CASE("dflash2 forward: an IDENTITY conv is BIT-IDENTICAL to no conv at all") { + // taps=1, base_kernel all ones, kernel_projection all zeros. In bf16 that is + // exactly `out = 1 * x`, with no tap to mask -- so a DFlash2 draft carrying it + // must reproduce the DFlash1 logits BIT-FOR-BIT. This is what separates + // "correctly placed and transparent" from "applied somewhere". + for (int64_t block : {int64_t{8}, int64_t{16}}) { + CAPTURE(block); + Dims base; + base.block = block; + Dims ident = base; + ident.conv_taps = 1; + const ScratchCkpt ck0(DraftEntries(base)); + const ScratchCkpt ck1(DraftEntries(ident)); + const std::vector without = Forward(LoadDraft(ck0, base, DraftConfig(base)), + DraftConfig(base), block); + const std::vector with = Forward(LoadDraft(ck1, ident, DraftConfig(ident)), + DraftConfig(ident), block); + REQUIRE(without.size() == with.size()); + CHECK(BitEqual(without, with)); + } +} + +TEST_CASE("dflash2 forward: the conv is LOAD-BEARING at block 8 and at block 16") { + // The reachability assertion. A real (non-identity) conv must move the draft + // logits at BOTH published block shapes: `z-lab/Qwen3.8-27B-DFlash2` ships + // block 8 and `z-lab/Muse-Glimmer-30B-DFlash2` ships block 16. Deleting the + // DflashConvPrepare/DflashConvFinish call sites in the layer body makes each + // comparison below EQUAL and this case red. + for (int64_t block : {int64_t{8}, int64_t{16}}) { + CAPTURE(block); + Dims base; + base.block = block; + Dims live = base; + live.conv_taps = 2; + live.attn_conv_active = true; + live.mlp_conv_active = true; + const ScratchCkpt ck0(DraftEntries(base)); + const ScratchCkpt ck1(DraftEntries(live)); + const std::vector without = Forward(LoadDraft(ck0, base, DraftConfig(base)), + DraftConfig(base), block); + const std::vector with = Forward(LoadDraft(ck1, live, DraftConfig(live)), + DraftConfig(live), block); + REQUIRE(without.size() == with.size()); + CHECK_FALSE(BitEqual(without, with)); + // And the CONTEXT-AWARE body -- a separate layer body, separately wired -- + // must move too. Its DFlash1 output is its own baseline, because the two + // bodies build the attention differently. + const std::vector ctx_without = + ForwardWithContext(LoadDraft(ck0, base, DraftConfig(base)), DraftConfig(base), block); + const std::vector ctx_with = + ForwardWithContext(LoadDraft(ck1, live, DraftConfig(live)), DraftConfig(live), block); + REQUIRE(ctx_without.size() == ctx_with.size()); + CHECK_FALSE(BitEqual(ctx_without, ctx_with)); + } +} + +TEST_CASE("dflash2 forward: the prepare side and the finish side land in DIFFERENT places") { + // `base_kernel` dim 0 is prepare/finish. Driving only side 0 convolves the + // sublayer INPUT; driving only side 1 convolves its OUTPUT. If the side index + // were ignored -- or if both calls read the same half -- these two would agree. + Dims prep; + prep.conv_taps = 2; + prep.attn_conv_active = true; + prep.mlp_conv_active = true; + prep.active_side = 0; + Dims fin = prep; + fin.active_side = 1; + const ScratchCkpt ck0(DraftEntries(prep)); + const ScratchCkpt ck1(DraftEntries(fin)); + const std::vector a = Forward(LoadDraft(ck0, prep, DraftConfig(prep)), + DraftConfig(prep), prep.block); + const std::vector b = Forward(LoadDraft(ck1, fin, DraftConfig(fin)), + DraftConfig(fin), fin.block); + REQUIRE(a.size() == b.size()); + CHECK_FALSE(BitEqual(a, b)); +} + +TEST_CASE("dflash2 forward: attention_conv and mlp_conv wrap DIFFERENT sublayers") { + // Two convs, one per sublayer, and upstream builds them with identical + // arguments -- so nothing but the CALL SITE distinguishes them. Wiring both to + // the attention (or both to the MLP) would make these two runs agree. + Dims attn; + attn.conv_taps = 2; + attn.attn_conv_active = true; + Dims mlp = attn; + mlp.attn_conv_active = false; + mlp.mlp_conv_active = true; + const ScratchCkpt ck0(DraftEntries(attn)); + const ScratchCkpt ck1(DraftEntries(mlp)); + const std::vector a = Forward(LoadDraft(ck0, attn, DraftConfig(attn)), + DraftConfig(attn), attn.block); + const std::vector b = Forward(LoadDraft(ck1, mlp, DraftConfig(mlp)), + DraftConfig(mlp), mlp.block); + REQUIRE(a.size() == b.size()); + CHECK_FALSE(BitEqual(a, b)); +} + +TEST_CASE("dflash2 forward: a ragged query block is REFUSED rather than mis-masked") { + // The conv masks its taps by `row index mod conv_block_size`, which is the + // intra-block offset only while every request block is contiguous and + // block-aligned. A ragged batch would mask the WRONG taps and be invisible: + // the verify is lossless, so only acceptance would move. + Dims dm; + dm.conv_taps = 2; + const ScratchCkpt ck(DraftEntries(dm)); + const HfConfig c = DraftConfig(dm); + const Qwen3DFlashWeights w = LoadDraft(ck, dm, c); + std::vector ids(12), pos(12); + for (int i = 0; i < 12; ++i) { + ids[static_cast(i)] = i % 8; + pos[static_cast(i)] = i; + } + const std::vector ragged = {0, 5, 12}; // neither block is 8 rows + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + CHECK_THROWS_WITH_AS(Qwen3DFlashModel::ForwardBlockLogits(ids, pos, ragged, w, c, q), + doctest::Contains("conv_block_size"), std::exception); +} + +TEST_CASE("dflash2 propose: the conv RUNS and THEN the selector refuses by name") { + // The ORDER is the claim. `DflashProposeBlock` is one of the two places that + // turn draft logits into draft tokens (the other is + // `GPUModelRunner::propose_drafts_block`), and it runs the draft block forward + // -- grouped convolution and all -- BEFORE anything samples. So a DFlash2 draft + // reaching here has already executed every line of W2, and what it is refused + // for is the candidate selector alone. + // + // Refusing before the forward would satisfy a "DFlash2 is refused" assertion + // while leaving the whole convolution unreachable from any production entry + // point, which is what .agents/reachability.md calls the test-only driver. + // Deleting the `RefuseDflash2CandidateSelector` call inside `DflashProposeBlock` + // turns this case red. + Dims dm; + dm.conv_taps = 2; + dm.attn_conv_active = true; + dm.mlp_conv_active = true; + const ScratchCkpt ck(DraftEntries(dm)); + const HfConfig c = DraftConfig(dm); + const Qwen3DFlashWeights w = LoadDraft(ck, dm, c); + REQUIRE(w.IsDflash2()); + + const int64_t T = dm.block; // one request, one (1+k) query block + std::vector ids(static_cast(T)), pos(static_cast(T)); + for (int64_t i = 0; i < T; ++i) { + ids[static_cast(i)] = static_cast(i % c.vocab_size); + pos[static_cast(i)] = static_cast(i); + } + const std::vector cu = {0, static_cast(T)}; + const std::vector ctx_cu = {0, 0}; + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::string what; + try { + (void)vllm::v1::DflashProposeBlock(w, c, {}, {}, ctx_cu, ids, pos, cu, + /*num_reqs=*/1, /*k=*/static_cast(T - 1), q); + FAIL("expected the candidate-selector refusal"); + } catch (const std::exception& e) { + what = e.what(); + } + INFO("what: ", what); + CHECK(what.find("CANDIDATE SELECTOR") != std::string::npos); + CHECK(what.find("convolution IS implemented") != std::string::npos); + CHECK(what.find("#1314") != std::string::npos); +} + +TEST_CASE("dflash1 propose: a DFlash1 draft still proposes through the same entry") { + // The instrument's precondition again: the refusal above must be about DFlash2 + // and not about `DflashProposeBlock`. The same call on a DFlash1 draft returns + // k tokens per request. + Dims dm; // conv_taps 0 + const ScratchCkpt ck(DraftEntries(dm)); + const HfConfig c = DraftConfig(dm); + const Qwen3DFlashWeights w = LoadDraft(ck, dm, c); + REQUIRE_FALSE(w.IsDflash2()); + const int64_t T = dm.block; + std::vector ids(static_cast(T)), pos(static_cast(T)); + for (int64_t i = 0; i < T; ++i) { + ids[static_cast(i)] = static_cast(i % c.vocab_size); + pos[static_cast(i)] = static_cast(i); + } + const std::vector cu = {0, static_cast(T)}; + const std::vector ctx_cu = {0, 0}; + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const vllm::v1::DflashProposeResult r = vllm::v1::DflashProposeBlock( + w, c, {}, {}, ctx_cu, ids, pos, cu, 1, static_cast(T - 1), q); + REQUIRE(r.draft_token_ids.size() == 1); + CHECK(r.draft_token_ids[0].size() == static_cast(T - 1)); +} + +TEST_CASE("dflash2 forward: prepare reads base_kernel[0] and finish reads base_kernel[1]") { + // `base_kernel` dim 0 is the SIDE. With taps=1 and a ZERO projection the conv + // is exactly `out = base[side] * x` in bf16, so the two sides are separable by + // a single scalar each and nothing else in the model changes. + // + // A base = (1, 1) -> the exact identity + // B base = (1, 2) -> only the FINISH side scales + // C base = (2, 1) -> only the PREPARE side scales + // + // A model that passed side 0 to both calls would make B the identity and B == A; + // one that passed side 1 to both would make C the identity. Comparing B and C + // to A pins BOTH directions, which comparing them only to each other does not. + Dims a; + a.conv_taps = 1; + Dims b = a; + b.base_side1 = 2.0f; + Dims cc = a; + cc.base_side0 = 2.0f; + const ScratchCkpt cka(DraftEntries(a)); + const ScratchCkpt ckb(DraftEntries(b)); + const ScratchCkpt ckc(DraftEntries(cc)); + const std::vector la = Forward(LoadDraft(cka, a, DraftConfig(a)), DraftConfig(a), a.block); + const std::vector lb = Forward(LoadDraft(ckb, b, DraftConfig(b)), DraftConfig(b), b.block); + const std::vector lc = Forward(LoadDraft(ckc, cc, DraftConfig(cc)), DraftConfig(cc), cc.block); + CHECK_FALSE(BitEqual(la, lb)); // the FINISH side must reach the model + CHECK_FALSE(BitEqual(la, lc)); // the PREPARE side must reach the model + CHECK_FALSE(BitEqual(lb, lc)); // and they must land in different places +} + +namespace { + +// The DEVICE-KV body -- `ForwardBlockLogitsWithDeviceKV`, which is what +// `GPUModelRunner::propose_drafts_block` calls in production -- over one request +// with a two-row context appended through the production append path. +std::vector ForwardDeviceKV(const Qwen3DFlashWeights& w, const HfConfig& c, int64_t T) { + std::vector ids(static_cast(T)), pos(static_cast(T)); + for (int64_t i = 0; i < T; ++i) { + ids[static_cast(i)] = static_cast(i % c.vocab_size); + pos[static_cast(i)] = static_cast(2 + i); + } + const std::vector cu = {0, static_cast(T)}; + const std::vector ctx_cu = {0, 2}; + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::vector ctx(static_cast(2 * c.hidden_size)); + for (size_t i = 0; i < ctx.size(); ++i) + ctx[i] = 0.2f * static_cast(std::sin(0.13 * static_cast(i) + 0.4)); + auto store = Qwen3DFlashModel::MakeDeviceKVStore(c, q); + Qwen3DFlashModel::AppendContextKVDevice(*store, ctx, {0, 1}, w, c, q); + std::vector stores = {store.get()}; + return Qwen3DFlashModel::ForwardBlockLogitsWithDeviceKV(stores, ctx_cu, ids, pos, cu, w, c, q); +} + +} // namespace + +TEST_CASE("dflash2 forward: EACH conv reaches EACH layer body separately") { + // Three bodies carry the DFlash draft's layer loop -- + // `ForwardBlockLogits` (context-free), `ForwardWithCtxKVDev` (context-aware, + // what `DflashProposeBlock` calls) and `ForwardPagedBody` (the paged store, + // what the runner's production decode path reaches through + // `ForwardBlockLogitsWithDeviceKV`) -- and each has its OWN four call sites. + // + // A case that activates BOTH convs cannot tell one missing call site from + // none: the other conv still moves the logits and the comparison still passes. + // That is exactly what a mutation deleting only the context-aware body's + // attention_conv proved. So each conv is driven ALONE, through each body. + struct Arm { + const char* name; + bool attn; + bool mlp; + }; + const Arm arms[] = {{"attention_conv only", true, false}, {"mlp_conv only", false, true}}; + for (const Arm& arm : arms) { + CAPTURE(arm.name); + Dims base; + Dims live = base; + live.conv_taps = 2; + live.attn_conv_active = arm.attn; + live.mlp_conv_active = arm.mlp; + const ScratchCkpt ck0(DraftEntries(base)); + const ScratchCkpt ck1(DraftEntries(live)); + const Qwen3DFlashWeights w0 = LoadDraft(ck0, base, DraftConfig(base)); + const Qwen3DFlashWeights w1 = LoadDraft(ck1, live, DraftConfig(live)); + const HfConfig c0 = DraftConfig(base); + const HfConfig c1 = DraftConfig(live); + CHECK_FALSE(BitEqual(Forward(w0, c0, base.block), Forward(w1, c1, live.block))); + CHECK_FALSE(BitEqual(ForwardWithContext(w0, c0, base.block), + ForwardWithContext(w1, c1, live.block))); + CHECK_FALSE(BitEqual(ForwardDeviceKV(w0, c0, base.block), + ForwardDeviceKV(w1, c1, live.block))); + } +} diff --git a/tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp b/tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp new file mode 100644 index 000000000..8a7057ecd --- /dev/null +++ b/tests/vllm/v1/spec_decode/test_dflash2_selector_refusal.cpp @@ -0,0 +1,105 @@ +// SPEC-DFLASH2 W2 (#1314) — the boundary W2 leaves the DFlash2 architecture at. +// +// A `DFlash2DraftModel` draft now LOADS and its block forward RUNS, grouped +// dynamic convolution and all. What it cannot do is CHOOSE: upstream replaces +// the DFlash1 per-slot argmax with a candidate selector -- keep the target +// head's top-K per slot, score adjacent transitions +// ` + unary[c]`, walk the best path from the verified +// anchor (`vllm/model_executor/models/qwen3_dflash2.py` `CandidateSelector` and +// `vllm/v1/worker/gpu/spec_decode/dflash2/speculator.py` @ +// vllm-project/vllm#52816 head `19c9351904df4c63042671bc67a866ca48dc7d6f`), and +// none of that is ported. +// +// WHY THIS IS A REFUSAL AND NOT A FALLBACK, which is the whole content of this +// file: `SampleDflashBlockDrafts` would SUCCEED on a DFlash2 block. It would +// return well-formed tokens, the verify would accept or reject them losslessly, +// the engine would emit the TARGET's tokens either way, and only ACCEPTANCE +// would fall. No token gate in this repository can see that. So the engine +// refuses by name instead, and this suite is what holds it to that. +// +// The refusal is placed AFTER the forward on purpose. The forward is implemented +// and gated (tests/vllm/models/test_qwen3_dflash2_draft.cpp, +// tests/vt/test_ops_dflash2_grouped_conv.cpp); the choice is not. Refusing +// before it would leave every line of W2 unreachable from any production entry +// point -- AGENTS.md `## Nothing lands dead`. +#include + +#include +#include + +#include "vllm/model_executor/models/qwen3_dflash.h" +#include "vllm/v1/worker/gpu/spec_decode/dflash/speculator.h" + +using vllm::Qwen3DFlashWeights; +using vllm::v1::RefuseDflash2CandidateSelector; + +namespace { + +Qwen3DFlashWeights Dflash1() { + Qwen3DFlashWeights w; + w.num_taps = 5; + w.mask_token_id = 248070; + w.draft_vocab_size = 8; + return w; // conv_taps 0 -> IsDflash2() false +} + +Qwen3DFlashWeights Dflash2() { + Qwen3DFlashWeights w = Dflash1(); + w.conv_taps = 2; // dflash_config.conv_kernel_size on both published drafts + w.conv_group_size = 16; // dflash_config.conv_group_size on both + w.conv_block_size = 8; + return w; +} + +} // namespace + +TEST_CASE("dflash2: the candidate selector is REFUSED BY NAME for a DFlash2 draft") { + std::string what; + try { + RefuseDflash2CandidateSelector(Dflash2()); + FAIL("expected a refusal for a DFlash2 draft"); + } catch (const std::exception& e) { + what = e.what(); + } + INFO("what: ", what); + // The mechanism that is missing, named -- not "DFlash2 is unsupported". + CHECK(what.find("CANDIDATE SELECTOR") != std::string::npos); + // The mechanism that is NOT missing, so a reader is not sent to reimplement it. + CHECK(what.find("convolution IS implemented") != std::string::npos); + // Why a fallback is inadmissible, which is the part a future agent needs. + CHECK(what.find("only") != std::string::npos); + CHECK(what.find("acceptance falls") != std::string::npos); + // Who owns the wiring. + CHECK(what.find("W3") != std::string::npos); + CHECK(what.find("SPEC-DFLASH2") != std::string::npos); + CHECK(what.find("#1314") != std::string::npos); +} + +TEST_CASE("dflash2: a DFlash1 draft passes the selector check untouched") { + // The instrument's own precondition. A check that refused EVERY dflash draft + // would satisfy the case above while killing the lane that ships, and that + // case's assertions could not tell the two apart. + CHECK_NOTHROW(RefuseDflash2CandidateSelector(Dflash1())); +} + +TEST_CASE("dflash2: the DFlash1 per-slot argmax still answers for a DFlash1 block") { + // The thing the selector replaces, unchanged: the refusal above must not have + // moved DFlash1's sampling. Two requests, k=2, draft_vocab=3; request 0's mask + // rows peak at ids 2 and 0, request 1's at 1 and 2. + const std::vector logits = { + 0.0f, 0.0f, 0.0f, // req 0 anchor (never sampled) + 0.1f, 0.2f, 0.9f, // req 0 mask 0 -> 2 + 0.7f, 0.3f, 0.1f, // req 0 mask 1 -> 0 + 0.0f, 0.0f, 0.0f, // req 1 anchor + 0.2f, 0.8f, 0.4f, // req 1 mask 0 -> 1 + 0.1f, 0.2f, 0.6f, // req 1 mask 1 -> 2 + }; + const std::vector> drafts = + vllm::v1::SampleDflashBlockDrafts(logits, /*num_reqs=*/2, /*k=*/2, /*draft_vocab=*/3); + REQUIRE(drafts.size() == 2); + REQUIRE(drafts[0].size() == 2); + CHECK(drafts[0][0] == 2); + CHECK(drafts[0][1] == 0); + CHECK(drafts[1][0] == 1); + CHECK(drafts[1][1] == 2); +} diff --git a/tests/vt/test_ops_dflash2_grouped_conv.cpp b/tests/vt/test_ops_dflash2_grouped_conv.cpp new file mode 100644 index 000000000..e4c2cf392 --- /dev/null +++ b/tests/vt/test_ops_dflash2_grouped_conv.cpp @@ -0,0 +1,385 @@ +// vllm.cpp original (vt runtime). Unit tests for vt::DFlashGroupedConv — the +// DFlash2 draft's GROUPED DYNAMIC DEPTHWISE CONVOLUTION (SPEC-DFLASH2 W2, +// #1314). +// +// BEYOND-PIN. The reference is `_grouped_conv` +// (vllm/model_executor/models/qwen3_dflash2.py @ vllm-project/vllm#52816 head +// `19c9351904df4c63042671bc67a866ca48dc7d6f`), and the sequential expectation +// below is upstream's OWN reference loop from +// `tests/v1/spec_decode/test_dflash2.py::test_grouped_conv_matches_reference` at +// that head, transcribed rather than reinvented: +// +// for position in range(block_size): +// for tap in range(min(taps, position + 1)): +// expected[:, position] += (base[tap] + delta[:, position, tap, :, None]) +// * hidden_blocks[:, position - tap] +// +// WHAT IS GATED, and why each case exists. +// +// * BOTH position-mask arms. Upstream special-cases a power-of-two block to +// `position & (block-1)` and otherwise uses `position % block`. Upstream's own +// parametrize covers 5 and 8; this file covers 5 (modulo arm), 8 and 16 — +// 8 and 16 because they are the blocks the two PUBLISHED checkpoints ship +// (`z-lab/Qwen3.8-27B-DFlash2` block 8, `z-lab/Muse-Glimmer-30B-DFlash2` +// block 16) and neither upstream parameter reaches 16. +// * The BLOCK BOUNDARY. A tap must contribute NOTHING across it. The dedicated +// case below feeds a row whose predecessor lives in the previous block and +// asserts the output ignores it — that zeroing is the whole reason this is a +// block convolution rather than a sequence one, and a token gate cannot see +// it (a draft that leaks across the boundary still emits the target's tokens +// and only loses acceptance). +// * The GROUP map. `delta` is per GROUP and `base` is per CHANNEL; a +// transposed or mis-divided `g(c) = c / group_size` still produces finite, +// plausible output. The case asserts two channels of the same group take the +// same delta and two channels of different groups do not. +// * The SIDE. `base_kernel` dim 0 is prepare/finish, NOT a tap. The case +// asserts side 1 reads base[1] and coefficients[:,1] and differs from side 0 +// on the same input. +// * CUDA == CPU, BIT-FOR-BIT. Every step of this op is elementwise with a +// rounding to the tensor dtype, exactly as upstream's bf16 chain materializes +// it, so there is no reduction-order freedom and no envelope to hide behind. +#include + +#include +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/ops.h" + +using vt::Backend; +using vt::Device; +using vt::DeviceType; +using vt::DFlashGroupedConvArgs; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +namespace { + +Device Cpu() { return Device{DeviceType::kCPU, 0}; } +Queue Q() { return Queue{Cpu(), nullptr}; } + +Tensor Contig(void* data, DType dt, Device dev, const std::vector& shape) { + Tensor t; + t.data = data; + t.dtype = dt; + t.device = dev; + t.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = t.rank - 1; i >= 0; --i) { + t.shape[i] = shape[static_cast(i)]; + t.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return t; +} +Tensor F32(std::vector& v, const std::vector& shape) { + return Contig(v.data(), DType::kF32, Cpu(), shape); +} + +DFlashGroupedConvArgs Args(int64_t block, int64_t taps, int64_t groups, int64_t gsize, + int64_t side) { + DFlashGroupedConvArgs a; + a.block_size = block; + a.taps = taps; + a.num_groups = groups; + a.group_size = gsize; + a.side = side; + return a; +} + +// Deterministic LCG in [-2,2); avoids divergence across libstdc++. +std::vector RandF32(size_t n, uint32_t seed) { + std::vector v(n); + uint32_t s = seed; + for (auto& x : v) { + s = s * 1664525u + 1013904223u; + x = (static_cast(s >> 8) / static_cast(1u << 24)) * 4.0f - 2.0f; + } + return v; +} + +// UPSTREAM's reference loop, transcribed from +// tests/v1/spec_decode/test_dflash2.py::test_grouped_conv_matches_reference @ the +// PR head. `hidden` is [batch*block, H] and every request block is contiguous and +// block-aligned, exactly as upstream's `hidden_states.unflatten` assumes. f32 +// arithmetic: the op rounds to the tensor dtype at each step, and on f32 that +// rounding is the identity, so this is the exact expectation for an f32 run. +std::vector Reference(const std::vector& hidden, const std::vector& delta, + const std::vector& base, int64_t batch, int64_t block, + int64_t taps, int64_t groups, int64_t gsize, int64_t sides, + int64_t side) { + const int64_t H = groups * gsize; + std::vector out(static_cast(batch * block * H), 0.0f); + for (int64_t b = 0; b < batch; ++b) { + for (int64_t pos = 0; pos < block; ++pos) { + const int64_t row = b * block + pos; + for (int64_t tap = 0; tap < taps && tap <= pos; ++tap) { + for (int64_t g = 0; g < groups; ++g) { + const size_t di = static_cast(((row * sides + side) * taps + tap) * groups + g); + for (int64_t j = 0; j < gsize; ++j) { + const int64_t c = g * gsize + j; + const size_t bi = static_cast((side * taps + tap) * H + c); + out[static_cast(row * H + c)] += + (base[bi] + delta[di]) * hidden[static_cast((row - tap) * H + c)]; + } + } + } + } + } + return out; +} + +// Drive the op over random f32 inputs at one shape and compare to Reference(). +void RunReferenceCase(int64_t batch, int64_t block, int64_t taps, int64_t groups, + int64_t gsize, int64_t sides, int64_t side, uint32_t seed) { + const int64_t H = groups * gsize; + const int64_t T = batch * block; + std::vector hidden = RandF32(static_cast(T * H), seed); + std::vector delta = RandF32(static_cast(T * sides * taps * groups), seed + 1); + std::vector base = RandF32(static_cast(sides * taps * H), seed + 2); + std::vector got(static_cast(T * H), 0.0f); + + Tensor tx = F32(hidden, {T, H}); + Tensor tc = F32(delta, {T, sides, taps, groups}); + Tensor tb = F32(base, {sides, taps, H}); + Tensor to = F32(got, {T, H}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, Args(block, taps, groups, gsize, side)); + + const std::vector want = + Reference(hidden, delta, base, batch, block, taps, groups, gsize, sides, side); + for (size_t i = 0; i < want.size(); ++i) { + INFO("index ", i); + CHECK(got[i] == doctest::Approx(want[i]).epsilon(1e-5)); + } +} + +} // namespace + +TEST_CASE("dflash2-grouped-conv matches upstream's sequential reference at block 5, 8 and 16") { + // Upstream's own parametrize shape (batch 3, taps 3, groups 4, group_size 2), + // at its two block parameters plus the second published checkpoint's 16. + // block 5 -> the `position % block` arm + // block 8 -> the `position & (block-1)` arm, and z-lab/Qwen3.8-27B-DFlash2 + // block 16 -> the same arm at the shape z-lab/Muse-Glimmer-30B-DFlash2 ships + RunReferenceCase(/*batch=*/3, /*block=*/5, /*taps=*/3, /*groups=*/4, /*gsize=*/2, + /*sides=*/1, /*side=*/0, /*seed=*/11); + RunReferenceCase(3, 8, 3, 4, 2, 1, 0, 22); + RunReferenceCase(3, 16, 3, 4, 2, 1, 0, 33); +} + +TEST_CASE("dflash2-grouped-conv matches the reference at BOTH published checkpoint shapes") { + // taps 2 and conv_group_size 16 are what both published DFlash2 configs + // declare. Hidden is reduced from 5120/6656 to keep the reference loop cheap; + // what the case pins is the taps/group/block triple and the 2-SIDE buffer the + // real `kernel_projection` produces (out = 2*taps*num_groups). + RunReferenceCase(/*batch=*/2, /*block=*/8, /*taps=*/2, /*groups=*/5, /*gsize=*/16, + /*sides=*/2, /*side=*/0, /*seed=*/44); + RunReferenceCase(2, 8, 2, 5, 16, 2, 1, 55); + RunReferenceCase(2, 16, 2, 6, 16, 2, 0, 66); + RunReferenceCase(2, 16, 2, 6, 16, 2, 1, 77); +} + +TEST_CASE("dflash2-grouped-conv RED: a tap contributes NOTHING across the block boundary") { + // Two blocks of 2 rows, taps=2, one group of one channel, base = 1 for both + // taps, delta = 0. x = [10, 20, 30, 40]. + // row 0 (pos 0): tap 1 masked -> 10 + // row 1 (pos 1): 20 + 10 -> 30 + // row 2 (pos 0): tap 1 MASKED -> 30 <-- x[1]=20 is in the PREVIOUS block + // row 3 (pos 1): 40 + 30 -> 70 + // Without the mask row 2 would read 20 and answer 50. That leak is invisible to + // a token gate: the verify is lossless, so only acceptance moves. + std::vector x = {10, 20, 30, 40}; + std::vector delta(4 * 1 * 2 * 1, 0.0f); + std::vector base = {1, 1}; + std::vector got(4, 0.0f); + Tensor tx = F32(x, {4, 1}); + Tensor tc = F32(delta, {4, 1, 2, 1}); + Tensor tb = F32(base, {1, 2, 1}); + Tensor to = F32(got, {4, 1}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, Args(/*block=*/2, /*taps=*/2, 1, 1, 0)); + CHECK(got[0] == doctest::Approx(10.0f)); + CHECK(got[1] == doctest::Approx(30.0f)); + CHECK(got[2] == doctest::Approx(30.0f)); // 50.0f iff the boundary leaks + CHECK(got[3] == doctest::Approx(70.0f)); +} + +TEST_CASE("dflash2-grouped-conv: delta is per GROUP and base is per CHANNEL") { + // One row, taps=1, 2 groups of 2 channels. base = [1,2,3,4] (per channel), + // delta = [10, 100] (per group). x = 1 everywhere. + // channel 0 (group 0): 1 + 10 = 11 + // channel 1 (group 0): 2 + 10 = 12 + // channel 2 (group 1): 3 + 100 = 103 + // channel 3 (group 1): 4 + 100 = 104 + // A transposed group map, or `g(c) = c % group_size`, answers a different + // permutation of the same four numbers and stays finite. + std::vector x = {1, 1, 1, 1}; + std::vector delta = {10, 100}; + std::vector base = {1, 2, 3, 4}; + std::vector got(4, 0.0f); + Tensor tx = F32(x, {1, 4}); + Tensor tc = F32(delta, {1, 1, 1, 2}); + Tensor tb = F32(base, {1, 1, 4}); + Tensor to = F32(got, {1, 4}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, Args(/*block=*/1, /*taps=*/1, /*groups=*/2, + /*gsize=*/2, 0)); + CHECK(got[0] == doctest::Approx(11.0f)); + CHECK(got[1] == doctest::Approx(12.0f)); + CHECK(got[2] == doctest::Approx(103.0f)); + CHECK(got[3] == doctest::Approx(104.0f)); +} + +TEST_CASE("dflash2-grouped-conv: base_kernel dim 0 is the SIDE, not a tap") { + // taps=1, 1 group, 1 channel, 2 SIDES. base = [[5],[7]], delta = [[0],[0]]. + // x = 2. Side 0 must answer 10 and side 1 must answer 14. Reading dim 0 as a + // tap would make side 1 unreachable and answer 10 twice — the shape defect the + // 27B header ruled out (base_kernel is (2, taps=2, 5120) with taps ALSO 2, so + // the two axes are indistinguishable by size on the real checkpoint). + std::vector x = {2}; + std::vector delta = {0, 0}; + std::vector base = {5, 7}; + std::vector got0(1, 0.0f), got1(1, 0.0f); + Tensor tx = F32(x, {1, 1}); + Tensor tc = F32(delta, {1, 2, 1, 1}); + Tensor tb = F32(base, {2, 1, 1}); + Tensor t0 = F32(got0, {1, 1}); + Tensor t1 = F32(got1, {1, 1}); + Queue q = Q(); + vt::DFlashGroupedConv(q, t0, tx, tc, tb, Args(1, 1, 1, 1, /*side=*/0)); + vt::DFlashGroupedConv(q, t1, tx, tc, tb, Args(1, 1, 1, 1, /*side=*/1)); + CHECK(got0[0] == doctest::Approx(10.0f)); + CHECK(got1[0] == doctest::Approx(14.0f)); +} + +// =========================================================================== +// CUDA parity. Unlike the attention ops, this one is elementwise with a rounding +// to the tensor dtype after each materialized step, so CPU and CUDA must agree +// BIT-FOR-BIT and the gate asserts equality rather than an envelope. +namespace { + +bool HasCuda() { + try { + vt::GetBackend(DeviceType::kCUDA); + return true; + } catch (const std::runtime_error&) { + return false; + } +} + +Device Gpu() { return Device{DeviceType::kCUDA, 0}; } + +struct QueueGuard { + Backend& b; + Queue q; + explicit QueueGuard(Backend& backend) : b(backend), q(backend.CreateQueue()) {} + ~QueueGuard() { b.DestroyQueue(q); } + QueueGuard(const QueueGuard&) = delete; + QueueGuard& operator=(const QueueGuard&) = delete; +}; + +class DeviceTensor { + public: + DeviceTensor(Backend& b, Queue& q, DType dt, const std::vector& shape, + const void* host = nullptr) + : b_(b) { + int64_t numel = 1; + for (auto s : shape) numel *= s; + bytes_ = static_cast(numel) * vt::SizeOf(dt); + p_ = b_.Alloc(bytes_ == 0 ? 1 : bytes_); + if (host != nullptr) b_.Copy(q, p_, host, bytes_); + t_ = Contig(p_, dt, Gpu(), shape); + } + ~DeviceTensor() { b_.Free(p_); } + DeviceTensor(const DeviceTensor&) = delete; + DeviceTensor& operator=(const DeviceTensor&) = delete; + Tensor& tensor() { return t_; } + void Download(Queue& q, void* dst) { + b_.Copy(q, dst, p_, bytes_); + b_.Synchronize(q); + } + + private: + Backend& b_; + void* p_ = nullptr; + size_t bytes_ = 0; + Tensor t_; +}; + +std::vector ToBf16(const std::vector& v) { + std::vector o(v.size()); + for (size_t i = 0; i < v.size(); ++i) o[i] = vt::F32ToBF16(v[i]); + return o; +} + +// One shape, run on CPU and CUDA in the SAME dtype, asserted BIT-EQUAL. +void RunCudaParity(int64_t batch, int64_t block, int64_t taps, int64_t groups, int64_t gsize, + int64_t sides, int64_t side, DType dt, uint32_t seed) { + const int64_t H = groups * gsize; + const int64_t T = batch * block; + const std::vector xf = RandF32(static_cast(T * H), seed); + const std::vector cf = RandF32(static_cast(T * sides * taps * groups), seed + 1); + const std::vector bf = RandF32(static_cast(sides * taps * H), seed + 2); + const DFlashGroupedConvArgs a = Args(block, taps, groups, gsize, side); + + std::vector xh, ch, bh; + const void* xp = xf.data(); + const void* cp = cf.data(); + const void* bp = bf.data(); + if (dt == DType::kBF16) { + xh = ToBf16(xf); + ch = ToBf16(cf); + bh = ToBf16(bf); + xp = xh.data(); + cp = ch.data(); + bp = bh.data(); + } + const size_t esz = vt::SizeOf(dt); + const size_t obytes = static_cast(T * H) * esz; + + std::vector cpu_out(obytes, 0); + { + Tensor tx = Contig(const_cast(xp), dt, Cpu(), {T, H}); + Tensor tc = Contig(const_cast(cp), dt, Cpu(), {T, sides, taps, groups}); + Tensor tb = Contig(const_cast(bp), dt, Cpu(), {sides, taps, H}); + Tensor to = Contig(cpu_out.data(), dt, Cpu(), {T, H}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, a); + } + + Backend& gpu = vt::GetBackend(DeviceType::kCUDA); + QueueGuard g(gpu); + DeviceTensor dx(gpu, g.q, dt, {T, H}, xp); + DeviceTensor dc(gpu, g.q, dt, {T, sides, taps, groups}, cp); + DeviceTensor db(gpu, g.q, dt, {sides, taps, H}, bp); + DeviceTensor dout(gpu, g.q, dt, {T, H}); + vt::DFlashGroupedConv(g.q, dout.tensor(), dx.tensor(), dc.tensor(), db.tensor(), a); + std::vector got(obytes, 0); + dout.Download(g.q, got.data()); + + CHECK(std::memcmp(got.data(), cpu_out.data(), obytes) == 0); +} + +} // namespace + +TEST_CASE("dflash2-grouped-conv CUDA is BIT-IDENTICAL to the CPU reference") { + if (!HasCuda()) { + MESSAGE("no CUDA backend; skipping CUDA dflash2-grouped-conv parity"); + return; + } + // Both published block shapes, in bf16 (what both checkpoints store) and f32, + // plus the modulo arm and both sides. + RunCudaParity(/*batch=*/4, /*block=*/8, /*taps=*/2, /*groups=*/320, /*gsize=*/16, + /*sides=*/2, /*side=*/0, DType::kBF16, 101); + RunCudaParity(4, 8, 2, 320, 16, 2, 1, DType::kBF16, 202); + RunCudaParity(4, 16, 2, 416, 16, 2, 0, DType::kBF16, 303); + RunCudaParity(4, 16, 2, 416, 16, 2, 1, DType::kBF16, 404); + RunCudaParity(3, 5, 3, 4, 2, 1, 0, DType::kF32, 505); + RunCudaParity(3, 8, 3, 4, 2, 1, 0, DType::kF32, 606); +} From 39af1ca9801f344a92e6ad646a9fe7d01e8e9ec7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 20 Aug 2026 08:18:19 +0000 Subject: [PATCH 2/3] fix(SPEC-DFLASH2): pin the per-step bf16 rounding, and correct what the selector refusal actually reaches (#1314) W2's fresh review returned FAIL on four findings. None was a defect in the shipped behaviour -- the kernel was verified line by line against upstream `_grouped_conv` at the PR head and the convolution is genuinely production-reached -- and all four were about what this repository CLAIMED to have proven. They are repaired here. **The wave's central numerics claim had no executing assertion.** W2 chose per-step bf16 rounding, and that choice is the whole basis for specifying the CUDA arm BIT-IDENTICAL rather than within a tolerance envelope. Nothing tested it. The reviewer replaced the bf16 branch of the `round` lambda in `src/vt/cpu/cpu_ops.cpp` with `return v;`, it compiled clean, and both focused suites stayed fully green: `test_ops_dflash2_grouped_conv` 6/6 cases, 9410/9410 assertions, `Status: SUCCESS!`; `test_qwen3_dflash2_draft` 16/16, 108/108, `SUCCESS!`. Three causes, each verified: the op suite drove the op only in f32, where `round()` is the IDENTITY by construction; `RunCudaParity` returns early on a host with no `nvcc`, so its bf16 shapes never execute; and the draft suite does execute the bf16 branch but asserts only RELATIONALLY between two runs of the same kernel, so a rounding change moves both arms together and cancels. Two CPU-only bf16 cases now pin it. One is hand-computed against literals, at taps 2 over two blocks with `delta = 2^-9` so all three rounding steps run: `bf16(3*89) = bf16(267) = 268` and then `bf16(268 + 3) = bf16(271) = 272`, where rounding once at the end answers 270. Six of its eight outputs move under the other policy. The other asserts bit-exactness at three shapes -- both published blocks, both sides, and taps 3 so more than one accumulate rounding chains -- against a reference that rounds where UPSTREAM materializes rather than where our kernel does. Red first, with the same mutation: 8 cases / 2 failed, 9930 assertions / 225 failed, `Status: FAILURE!`, compile rc 0, `git diff --stat` showing the hunk. Restored byte-for-byte (sha256 verified), rebuilt, green: 8/8, 9930/9930, `SUCCESS!`. `include/vt/ops.h` said "and the gate asserts that" about the bit-identity; it now separates the half that is pinned from the half that is not. **The staged-slice disclosure understated the gap, and that disclosure is what AGENTS.md's permission rests on.** `## Owed` O7 and two comments said `RefuseDflash2CandidateSelector` "has TWO production call sites", one of them gated. It has ONE. `DflashProposeBlock` has no caller outside `tests/` -- an exhaustive grep finds only its definition, its declaration, two prose comments in `runner.cpp`, and tests -- so the site a test can delete-and-redden is test-only, while deleting the real site at `runner.cpp` leaves all four focused suites GREEN. Production coverage of the refusal is ZERO, not one of two. Gating the real site was preferred and is not reachable here: `propose_drafts_dflash` returns early unless `dflash_weights_` is set, that member is only set on the `LoadedModel` construction path, and the synthetic-weights `GPUModelRunner` constructors take no `SpeculativeConfig` at all, so a gate needs an on-disk target plus draft driven through the loader -- the harness O5 already waits on and W4 builds. O7 now says zero, names why, and names W4. **A recorded sha256 did not hash what was embedded.** The three `config.json` documents in `tests/vllm/models/test_qwen3_dflash2_draft.cpp` were labelled VERBATIM with a sha256 beside each. Two were the published file minus its trailing newline, and the third, `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash`, had been re-indented to two spaces and had lost its `auto_map` key, so its recorded hash described a file that is not in this repository. All three literals are now the published bytes, re-fetched and re-hashed, and each recorded sha256 hashes the literal beside it. **The admitted checkpoints were not pinned.** `docs/USAGE.md` gains `## DFlash2 drafts: the exact checkpoints`: repo, revision, file, byte count and sha256 for the admitted bf16 draft (`z-lab/Qwen3.8-27B-DFlash2` @ `50307d4c`, 3 848 817 896 bytes) and for all three refused GGUF arms, plus the target it heads and the second published draft's revision. Every hash was computed over a local copy rather than read from a hub API, which can return an `lfs.oid` that hashes nothing, and the shard was checked semantically too: 81 tensors, all BF16, last data offset exactly on the file size. **One reviewer concern is recorded and NOT fixed.** New `## Owed` O8: `PrecomputeContextKVDevice` projects every layer's context K/V from one shared `hidden_norm(context_states)` and applies no convolution, while upstream has no analogue -- its context K/V is what earlier block forwards wrote, which under DFlash2 came from a conv'd stream. Whether the shortcut stays equivalent now that the conv exists is unaddressed, and if it is wrong the symptom is acceptance-only and token-invisible, which is this row's own named defect class. Owner W3/W4 under [#1314](https://github.com/mudler/vllm.cpp/issues/1314). O6 also gains what it did not say: the bf16 arm's CPU == CUDA bit-identity is still unpinned on BOTH sides, because that case has never compiled here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/dflash2-spec-decode.md | 144 ++++++++++++--- docs/USAGE.md | 53 ++++++ .../gpu/spec_decode/dflash/speculator.h | 14 +- include/vt/ops.h | 13 +- .../vllm/models/test_qwen3_dflash2_draft.cpp | 76 ++++++-- tests/vt/test_ops_dflash2_grouped_conv.cpp | 168 +++++++++++++++++- 6 files changed, 412 insertions(+), 56 deletions(-) diff --git a/.agents/specs/dflash2-spec-decode.md b/.agents/specs/dflash2-spec-decode.md index 7a8e0f8ab..7624b0d0b 100644 --- a/.agents/specs/dflash2-spec-decode.md +++ b/.agents/specs/dflash2-spec-decode.md @@ -464,24 +464,98 @@ list items. (`tests/vt/test_ops_dflash2_grouped_conv.cpp`, six shapes covering both published blocks in bf16 and the modulo arm in f32). It has NEVER COMPILED: the authoring host has no `nvcc`, so the CUDA case reports - `no CUDA backend; skipping CUDA dflash2-grouped-conv parity` and the file's - 9410 assertions are all CPU. Two specific things are unproven rather than + `no CUDA backend; skipping CUDA dflash2-grouped-conv parity` and every one of + the file's assertions runs on CPU. Two specific things are unproven rather than merely unrun: that the kernel compiles at all, and that `__fadd_rn`/`__fmul_rn` plus `ResRound` reproduce the CPU reference BIT-FOR-BIT on the f32 arm, where the intrinsics are the only thing forbidding an FMA contraction the CPU build pins off. This is named here rather than reported as a pass. -- **O7 — the runner's own selector-refusal call site is not gated.** Owner: this + + **What the wave's second fresh review corrected here.** The sentence above used + to read "the file's 9410 assertions are all CPU". That was true and still read + as coverage the file did not have: all 9410 were also f32, and on f32 the + kernel's per-step rounding is the IDENTITY by construction. So the wave's + central numerics claim — per-step rounding, which is the entire reason the CUDA + arm is specified BIT-IDENTICAL rather than within an envelope — had no + executing assertion on either side. The reviewer proved it rather than read it: + replacing the bf16 branch of the `round` lambda in `src/vt/cpu/cpu_ops.cpp` + with `return v;` compiled clean and left BOTH focused suites fully green + (`test_ops_dflash2_grouped_conv` 6/6 cases, 9410/9410 assertions, `SUCCESS!`; + `test_qwen3_dflash2_draft` 16/16, 108/108, `SUCCESS!`). The draft suite does + execute the bf16 branch, but every assertion in it is RELATIONAL between two + runs of the same kernel, so a rounding-policy change moves both arms together + and cancels. + + The CPU half is now pinned: two CPU-only bf16 cases, one hand-computed against + literals that differ from the round-once-at-the-end answer in six of eight + outputs, and one bit-exact at three shapes against a reference that rounds + where UPSTREAM materializes rather than where our kernel does. Under the same + mutation they fail: 8 cases / 2 failed, 9930 assertions / 225 failed, + `Status: FAILURE!`. What is still owed is unchanged in kind and smaller in + size: CPU == CUDA bit-identity remains unpinned on BOTH sides, because the CUDA + arm has still never compiled or run here. +- **O7 — NO production call site of the selector refusal is gated.** Owner: this row, discharged by W4. Issue [#1314](https://github.com/mudler/vllm.cpp/issues/1314). - `RefuseDflash2CandidateSelector` has TWO production call sites: - `DflashProposeBlock` (gated — deleting the call turns - `test_qwen3_dflash2_draft` red, 1 case / 1 assertion) and - `GPUModelRunner::propose_drafts_block` (NOT gated: entering it needs a - constructed `GPUModelRunner` with a loaded target, a KV cache and a spec - config). The two call sites are one line apart in intent and are easy to keep - in step, and the ungated one is the one a user actually arrives through. W4 - wires the DFlash2 speculator and is the wave that can enter that path. + `RefuseDflash2CandidateSelector` is called from exactly two places, and only + one of them is production: + + - `GPUModelRunner::propose_drafts_block` (`src/vllm/v1/worker/gpu/runner.cpp`) + — the PRODUCTION site, and NOT gated. Mutation-proven: deleting this call + leaves all four focused suites GREEN. + - `DflashProposeBlock` + (`src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp`) — gated + (deleting the call turns `test_qwen3_dflash2_draft` red, 1 case), but + `DflashProposeBlock` has NO caller outside `tests/` at this commit. An + exhaustive grep finds only its definition, its declaration, two prose + comments in `runner.cpp`, and tests. This site is TEST-ONLY. + + W2's own record used to say the refusal "has TWO production call sites", one of + them gated. That was wrong in the direction that flatters: production coverage + of the refusal is ZERO, not one of two. AGENTS.md `## Nothing lands dead` + grants the staged-slice exception only when this list names what is unreached, + so an inaccurate entry here is a defect in the permission and not a wording + problem. It is corrected rather than annotated. + + **Why it is not gated where it was found.** Reaching the runner site means + reaching `propose_drafts_dflash`, which returns early unless `dflash_weights_` + is set, and that member is only ever set on the `LoadedModel` construction + path; the synthetic-weights `GPUModelRunner` constructors take no + `SpeculativeConfig` at all. A gate therefore needs an on-disk TARGET plus an + on-disk draft driven through the loader, a step that captures the target's aux + multi-tap, and a populated per-request device KV store. That is the harness O5 + is already waiting on, and the one W4 builds when it wires the DFlash2 + speculator. + + **What this does NOT put in doubt.** The grouped convolution is production- + reached and mutation-detected. Each conv call site was deleted separately — + `attention_conv` and `mlp_conv`, in each of the three layer bodies, six + mutations over twelve calls — and every one turned `test_qwen3_dflash2_draft` + red, including through `ForwardPagedBody`, which is what + `ForwardBlockLogitsWithDeviceKV` and therefore the production decode path + reaches. +- **O8 — the context-KV precompute applies NO convolution, and nothing says + whether that stays equivalent.** Owner: this row, answered by W3 or W4. Issue + [#1314](https://github.com/mudler/vllm.cpp/issues/1314). Raised by the wave's + second fresh reviewer, and recorded rather than fixed because answering it is a + design question about the engine's context path rather than a repair to what W2 + shipped. + + `Qwen3DFlashModel::PrecomputeContextKVDevice` + (`src/vllm/model_executor/models/qwen3_dflash.cpp:149`) projects EVERY layer's + context K/V from one shared `hidden_norm(context_states)`, and applies no + convolution at any layer. Upstream has no analogue of this precompute: its + context K/V is whatever the earlier block forwards wrote, and under DFlash2 + those forwards wrote from a CONV'd stream. Ours is projected from an + unconvolved shared tensor; upstream's came from a convolved per-layer one. + + The shortcut predates DFlash2 and is correct for DFlash1, where no conv exists. + Whether it stays equivalent now that the conv does is not addressed anywhere in + this spec. If it is wrong, the symptom is the defect class this row exists to + remove: acceptance-only and token-invisible, because the verify is lossless and + the engine still emits the target's tokens. W3 touches this path when it lands + the selector, and either shows the two agree or replaces the precompute. ## Now @@ -496,9 +570,11 @@ is the project's first grouped dynamic depthwise convolution: `base_kernel` dim 0 the prepare/finish SIDE rather than a tap. The CPU kernel is the authoritative reference and rounds to the tensor dtype after each step, as upstream's bf16 chain materializes it — which is what lets the CUDA mirror be -asserted BIT-IDENTICAL rather than within an envelope. Both of upstream's -position-mask arms are ported (`pos & (block-1)` and `pos % block`) and gated at -block 5, 8 and 16. +specified BIT-IDENTICAL rather than within an envelope. That per-step rounding +is now itself gated, on CPU and in bf16, which is the only arm where it is +observable at all: see O6 for what the wave's second review found and what it +cost. Both of upstream's position-mask arms are ported (`pos & (block-1)` and +`pos % block`) and gated at block 5, 8 and 16. **The refusal MOVED so that the conv could be reached.** A safetensors `DFlash2DraftModel` draft is now admitted at `CheckDflash2DraftArm`, loads its @@ -519,20 +595,29 @@ none: it activated both convs at once, so deleting only the context-aware body's running the mutation rather than by reading the test, and both were repaired before the wave landed — each conv is now driven ALONE through each body, and the two sides are separated by `base_kernel[side]` scalars against a common identity -baseline. The final mutation set turns the focused suites red for: each body's -call sites (three separate mutations), the side index, the block mask, the group -map, the `rope_parameters` fallback, the `dflash_config.block_size` fallback, the -`layer_types` fallback, the `attention_sink_bias` refusal, the uniform-block -guard, the `DflashProposeBlock` refusal call, and restoring W1's startup refusal. - -**Four `## Owed` entries are discharged and three are new.** O1 (the `is_causal` +baseline. It cost a third repair after the wave's second review, on the same +pattern: the PER-STEP ROUNDING had no executing assertion, because every case in +the op suite ran in f32 where that rounding is the identity (O6). The final +mutation set turns the focused suites red for: each body's call sites (three +separate mutations), the side index, the block mask, the group map, the per-step +bf16 rounding, the `rope_parameters` fallback, the `dflash_config.block_size` +fallback, the `layer_types` fallback, the `attention_sink_bias` refusal, the +uniform-block guard, the `DflashProposeBlock` refusal call, and restoring W1's +startup refusal. It does NOT turn them red for the refusal's production call +site, which is O7. + +**Four `## Owed` entries are discharged and four are new.** O1 (the `is_causal` rule was inert), O2's weight half, O3 (`MakeQwen3DFlashDraftConfig` could not parse either published DFlash2 config) and O4 (`layer_types`, plus the `attention_sink_bias` refusal that had to land with it) are closed. O5 records that `LoadDflashDraft`'s own `conv_block_size = k + 1` is UNGATED and mutation-proven so; O6 records that the CUDA arm has never compiled on this host -and is owed to a GPU lease; O7 records that the runner's selector-refusal call -site is not gated. None of the three is a claim wearing a pass. +and is owed to a GPU lease, and what the CPU-side rounding gate does and does not +now prove; O7 records that NO production call site of the selector refusal is +gated — zero, not one of two, which is what the entry said before the wave's +second review measured it; O8 records that the context-KV precompute applies no +convolution and that nobody has shown the shortcut stays equivalent now that the +convolution exists. None of the four is a claim wearing a pass. **#1327 is corrected in this wave.** `## Upstream chain` said no published checkpoint exercised `input_embedding_scale`, `output_multiplier` or @@ -546,6 +631,19 @@ declare `model_type` `qwen3`), `## Gates` G1 now requires both block shapes, and `## Risks/decisions` D9 records that the scalars must be gated against the checkpoint that sets them rather than against defaults. +**The admitted checkpoints are now PINNED, and their hashes are ours.** +`docs/USAGE.md` gains a `## DFlash2 drafts: the exact checkpoints` table: repo, +revision, file, byte count and sha256 for the admitted bf16 safetensors draft +(`z-lab/Qwen3.8-27B-DFlash2` @ `50307d4c`) and for all three refused GGUF arms, +plus the target the draft heads and the second published draft's revision. Every +sha256 was computed over a local copy rather than read from a hub API, because an +unauthenticated tree API can return an `lfs.oid` that hashes nothing; the +safetensors shard was also checked semantically (81 tensors, all BF16, last data +offset exactly on the file size). The same section says what the gate actually +reads, which is not those bytes: the published `config.json` documents embedded +byte-for-byte in the test, and a safetensors file the test WRITES with the +published tensor names. + Next action: W3, the candidate selector — the lattice op, the codebooks in the loader, and the top-k that EMITS pairs (D2). It is the wave that lifts the refusal W2 leaves behind, and D9 binds it to Muse Glimmer's scalars. diff --git a/docs/USAGE.md b/docs/USAGE.md index 0a5098fa1..9ac77486f 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2843,6 +2843,59 @@ takes the GGUF branch above the hoist, which carries its own named refusal for a GGUF DSpark target (`SPEC-DSPARK`). Either way the draft is refused and nothing loads it as a Qwen3 draft. +## DFlash2 drafts: the exact checkpoints + +A DFlash2 draft is a SEPARATE checkpoint named by the `model` key of +`--speculative-config`, and it heads one specific target. A repo id alone is not +a pin, because a checkpoint can be re-quantized in place under an unchanged name, +so the revision is part of the identity. + +**Read what these weights currently buy you before you download 3.6 GiB.** A +safetensors `DFlash2DraftModel` draft is admitted as far as its CONVOLUTION and +no further: it loads, it runs the grouped dynamic depthwise convolution around +every attention and every MLP sublayer of every draft layer, and it is then +refused BY NAME at the candidate selector, which this engine does not implement +yet (`SPEC-DFLASH2`, [#1314](https://github.com/mudler/vllm.cpp/issues/1314)). +A startup notice says so, so the refusal at the first generated token is not a +surprise. These are therefore the checkpoints the port was BUILT and READ +against, not checkpoints that produce a draft token here today. + +| Arm | Repo and revision | File | Bytes | sha256 | +|---|---|---|---|---| +| Draft, bf16 safetensors — ADMITTED to the convolution | `z-lab/Qwen3.8-27B-DFlash2` @ `50307d4c4cde6860d4eee73e2547cd786fe8e8a4` | `model.safetensors` | 3 848 817 896 | `67fc76d68dc5a9415511a4f394ef744d67510cd20e93b37cc2cc7d28e4bab65c` | +| Draft, GGUF — REFUSED at startup | `z-lab/Qwen3.8-27B-DFlash2-GGUF` @ `57ab3265056d4024870b0621cfc2c127537020ed` | `Qwen3.8-27B-DFlash2-BF16.gguf` | 3 860 293 152 | `26af33a15b21475d668e4ee55639beea49932e7360b1144c6282721bcd127c14` | +| Draft, GGUF — REFUSED at startup | same | `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2 056 414 752 | `7f1c9a31a6ed40044c69f6508b50fd63b87abd8e1fb7fe4290303df549153751` | +| Draft, GGUF — REFUSED at startup | same | `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1 143 006 752 | `18a380efc9b7ed8d88677fc895f5c11ae170653434ee378f7348f715c14d0594` | +| Target the draft heads | `Qwen/Qwen3.8-27B` @ `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` | published shards | — | — | + +Every sha256 above was computed over the local copy on 2026-08-20, not read from +a hub API: an unauthenticated tree API can return an `lfs.oid` that is not a +hash of anything. Each file's size matches what the hub reports, and the +safetensors shard was checked semantically as well — 81 tensors, every one BF16, +and its last data offset lands exactly on the file size. + +**The GGUF rows are the REFUSED arm, and they are listed so the refusal is +checkable.** A GGUF DFlash2 drafter is refused at startup because its weight path +does not exist yet; it is classified by METADATA rather than by an architecture, +because a GGUF declares no architectures and the published DFlash2 GGUF writes +the same `dflash` architecture a DFlash1 one does. A file carrying +`dflash.selector_rank`, `dflash.selector_top_k` or `dflash.conv_kernel_size` is +refused, and a DFlash1 GGUF, which carries none of them, loads as before. The +GGUF drafter arm is a later wave of the row. + +**What the gate actually reads, which is not these bytes.** The published +`config.json` of `z-lab/Qwen3.8-27B-DFlash2` and of the second published DFlash2 +draft, `z-lab/Muse-Glimmer-30B-DFlash2` @ +`b54ffdd11fa9cfe2af370012e5763d492c904128`, are embedded BYTE-FOR-BYTE in +`tests/vllm/models/test_qwen3_dflash2_draft.cpp` with their sha256 recorded, and +the gate drives those documents through the production config builder. The +weight-loading cases run over a safetensors file the test WRITES, carrying the +published tensor names and shapes (`layers.N.attention_conv.base_kernel` bf16 +`[2, taps, hidden]`, `layers.N.mlp_conv.kernel_projection.weight`), because a +3.6 GiB download cannot be a unit-gate dependency. The two published drafts +differ in ways the gate needs: block 8 against block 16, and Muse Glimmer sets +`output_multiplier` and `final_logit_softcapping` where the 27B defaults them. + ## Muse Glimmer 30B from a GGUF k-quant The text tower loads from a `muse-glimmer`-architecture GGUF, so the 30B model diff --git a/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h b/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h index 8eeeb8e6a..b72a2e5e9 100644 --- a/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h +++ b/include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h @@ -61,10 +61,16 @@ namespace vllm::v1 { // why this is a refusal and not a fallback, and why it is placed AFTER the // forward: the forward is implemented and gated, the choice is not. // -// Called from the production draft step (`GPUModelRunner::propose_drafts_block`, -// src/vllm/v1/worker/gpu/runner.cpp) and from `DflashProposeBlock` below, which -// are the only two places that turn draft logits into draft tokens. Owed by W3 of -// the row. +// Two call sites turn draft logits into draft tokens and both refuse here: +// `GPUModelRunner::propose_drafts_block` (src/vllm/v1/worker/gpu/runner.cpp) and +// `DflashProposeBlock` below. Only the FIRST is production. `DflashProposeBlock` +// has no caller outside `tests/` at this commit -- grep it -- so the refusal that +// a test can delete-and-redden is the test-reachable one, and the site a user +// actually arrives through is UNGATED. Entering it needs a runner whose +// `dflash_weights_` is set, which only the `LoadedModel` construction path does, +// so a gate on it needs an on-disk target plus draft driven through the loader. +// That harness is W4's. See `## Owed` O7 of +// `.agents/specs/dflash2-spec-decode.md`; the refusal itself is owed by W3. void RefuseDflash2CandidateSelector(const Qwen3DFlashWeights& weights); // Greedy per-request draft pick over the (1+k) block logits — the greedy branch of diff --git a/include/vt/ops.h b/include/vt/ops.h index 63a05a3c9..83551ee22 100644 --- a/include/vt/ops.h +++ b/include/vt/ops.h @@ -806,7 +806,18 @@ struct DFlashPagedBlockAttentionArgs { // step, because upstream's chain materializes bf16 tensors at each one // (`base + delta`, `coefficients * blocks`, `output += ...`). This is elementwise // with no reduction-order freedom, so the CPU reference and the CUDA kernel are -// BIT-IDENTICAL rather than within an envelope, and the gate asserts that. +// specified BIT-IDENTICAL rather than within an envelope. +// +// WHAT IS ACTUALLY GATED, because the two halves of that sentence are not +// equally proven. The per-step POLICY is pinned on CPU in bf16 by +// `tests/vt/test_ops_dflash2_grouped_conv.cpp` — one hand-computed case with +// literal expectations that differ from the rounded-once-at-the-end answer in +// six of eight outputs, plus three shapes asserted bit-exact against a reference +// that rounds where UPSTREAM materializes. On f32 this rounding is the identity +// by construction, so no f32 case can see it and none is claimed to. The CPU == +// CUDA half is NOT proven: that case exists and is written to run, but it has +// never compiled on a host without `nvcc` and reports `no CUDA backend; +// skipping`. See `## Owed` O6 of `.agents/specs/dflash2-spec-decode.md`. struct DFlashGroupedConvArgs { int64_t block_size = 0; // 1 + num_speculative_tokens (the query block) int64_t taps = 0; // dflash_config.conv_kernel_size diff --git a/tests/vllm/models/test_qwen3_dflash2_draft.cpp b/tests/vllm/models/test_qwen3_dflash2_draft.cpp index ea2bf6b1a..54a4c4732 100644 --- a/tests/vllm/models/test_qwen3_dflash2_draft.cpp +++ b/tests/vllm/models/test_qwen3_dflash2_draft.cpp @@ -15,8 +15,9 @@ // `c.at("layer_types")`, which `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` does not // declare, while upstream reads `getattr(config, "layer_types", None)` // (`qwen3_dflash.py:134` and `:66` @ that head). The configs embedded below are -// the PUBLISHED files verbatim, with their sha256 recorded, so the gate does not -// depend on a checkout being present. +// the PUBLISHED files BYTE-FOR-BYTE, each with the sha256 OF THE EMBEDDED +// LITERAL recorded beside it, so the gate does not depend on a checkout being +// present and the recorded hash describes what the compiler actually sees. // // The `attention_sink_bias` refusal is the other half of O4 and is not // bookkeeping. Upstream reads `dflash_config.attention_sink_bias` and passes a @@ -71,6 +72,12 @@ namespace { // 2026-08-19). Kept whole rather than reduced: what this case gates is that the // PUBLISHED document parses, and a reduced copy would only prove that a document // this test wrote parses. +// +// The sha256 hashes THE LITERAL BELOW, trailing newline included, so the claim +// is checkable from this file alone rather than only against a copy on a share. +// The three literals in this file were all re-fetched and re-hashed on +// 2026-08-20; one of them (`kMimoDflashConfig`) had been re-indented and had +// lost a key, and it is now the published bytes. constexpr const char* kQwen38Dflash2Config = R"JSON({ "architectures": [ "DFlash2DraftModel" @@ -127,11 +134,14 @@ constexpr const char* kQwen38Dflash2Config = R"JSON({ "use_cache": true, "use_sliding_window": true, "vocab_size": 248320 -})JSON"; +} +)JSON"; -// `z-lab/Muse-Glimmer-30B-DFlash2`, config.json VERBATIM (sha256 -// cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9, read on -// 2026-08-19). The SECOND published DFlash2 checkpoint, and the one that makes +// `z-lab/Muse-Glimmer-30B-DFlash2` @ `b54ffdd11fa9cfe2af370012e5763d492c904128`, +// config.json VERBATIM (1326 bytes, sha256 +// cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9 over the +// literal below, read on 2026-08-19 and re-verified 2026-08-20). The SECOND +// published DFlash2 checkpoint, and the one that makes // #1327 a correction rather than a note: `block_size` 16 against the 27B's 8, // and `output_multiplier`/`final_logit_softcapping` SET rather than defaulted. constexpr const char* kMuseGlimmerDflash2Config = R"JSON({ @@ -192,16 +202,30 @@ constexpr const char* kMuseGlimmerDflash2Config = R"JSON({ "use_cache": false, "use_sliding_window": true, "vocab_size": 202048 -})JSON"; - -// `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` `dflash/config.json` VERBATIM (sha256 -// 2ed5a998f5f57e00a9fe14d2b3e767f06e49462a97eb09d80c927e112a585c9e, read on -// 2026-08-19). A DFlash1 draft, present here for O4: it is the ONLY published -// draft that declares no `layer_types`, and it is also the only one that -// declares `attention_sink_bias`. +} +)JSON"; + +// `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` @ +// `b754e6c86008bdb5cc901308dda5a38173ec7276`, `dflash/config.json` VERBATIM +// (1251 bytes, sha256 +// 2ed5a998f5f57e00a9fe14d2b3e767f06e49462a97eb09d80c927e112a585c9e over the +// literal below, re-fetched 2026-08-20). A DFlash1 draft, present here for O4: +// it is the ONLY published draft that declares no `layer_types`, and it is also +// the only one that declares `attention_sink_bias`. +// +// This copy was NOT verbatim when W2 landed. It had been re-indented to two +// spaces and had dropped `auto_map`, so the recorded sha256 hashed a file that +// was not in this repository and nothing here hashed what the test parsed. The +// published bytes are in, and the two facts the case turns on -- no +// `layer_types`, `attention_sink_bias` present -- are unchanged by the repair. constexpr const char* kMimoDflashConfig = R"JSON({ - "architectures": ["DFlashDraftModel"], + "architectures": [ + "DFlashDraftModel" + ], "model_type": "qwen3", + "auto_map": { + "AutoModel": "dflash.DFlashDraftModel" + }, "hidden_size": 6144, "intermediate_size": 16384, "num_hidden_layers": 5, @@ -212,7 +236,13 @@ constexpr const char* kMimoDflashConfig = R"JSON({ "partial_rotary_factor": 0.5, "block_size": 8, "dflash_config": { - "target_layer_ids": [0, 15, 31, 47, 69], + "target_layer_ids": [ + 0, + 15, + 31, + 47, + 69 + ], "mask_token_id": 151669, "num_anchors": 4096, "block_size": 8, @@ -781,10 +811,8 @@ TEST_CASE("dflash2 forward: a ragged query block is REFUSED rather than mis-mask } TEST_CASE("dflash2 propose: the conv RUNS and THEN the selector refuses by name") { - // The ORDER is the claim. `DflashProposeBlock` is one of the two places that - // turn draft logits into draft tokens (the other is - // `GPUModelRunner::propose_drafts_block`), and it runs the draft block forward - // -- grouped convolution and all -- BEFORE anything samples. So a DFlash2 draft + // The ORDER is the claim. `DflashProposeBlock` runs the draft block forward -- + // grouped convolution and all -- BEFORE anything samples. So a DFlash2 draft // reaching here has already executed every line of W2, and what it is refused // for is the candidate selector alone. // @@ -793,6 +821,16 @@ TEST_CASE("dflash2 propose: the conv RUNS and THEN the selector refuses by name" // point, which is what .agents/reachability.md calls the test-only driver. // Deleting the `RefuseDflash2CandidateSelector` call inside `DflashProposeBlock` // turns this case red. + // + // WHAT THIS CASE DOES NOT PROVE, stated because W2's first review found the + // claim overstated here: `DflashProposeBlock` itself has NO caller outside + // `tests/` at this commit, so the refusal call this case gates is the + // test-reachable one. The refusal a user arrives through is the identical call + // in `GPUModelRunner::propose_drafts_block`, and deleting THAT one leaves every + // suite in this repository green. It is `## Owed` O7 and it belongs to W4. The + // CONVOLUTION is a different matter and is genuinely production-reached: see + // the ForwardBlockLogitsWithDeviceKV cases below, whose call sites redden this + // suite one at a time. Dims dm; dm.conv_taps = 2; dm.attn_conv_active = true; diff --git a/tests/vt/test_ops_dflash2_grouped_conv.cpp b/tests/vt/test_ops_dflash2_grouped_conv.cpp index e4c2cf392..becd474c4 100644 --- a/tests/vt/test_ops_dflash2_grouped_conv.cpp +++ b/tests/vt/test_ops_dflash2_grouped_conv.cpp @@ -35,9 +35,18 @@ // * The SIDE. `base_kernel` dim 0 is prepare/finish, NOT a tap. The case // asserts side 1 reads base[1] and coefficients[:,1] and differs from side 0 // on the same input. -// * CUDA == CPU, BIT-FOR-BIT. Every step of this op is elementwise with a -// rounding to the tensor dtype, exactly as upstream's bf16 chain materializes -// it, so there is no reduction-order freedom and no envelope to hide behind. +// * THE ROUNDING POLICY, in bf16 and on CPU. Every step of this op rounds to +// the tensor dtype, exactly as upstream's bf16 chain materializes it. The +// cases above the CUDA section run in f32, where that rounding is the +// IDENTITY by construction and therefore invisible; the two bf16 cases pin it +// directly, one with hand-computed literals that differ from the +// round-once-at-the-end answer and one bit-exact against a reference that +// rounds where upstream materializes. +// * CUDA == CPU, BIT-FOR-BIT. The consequence of the policy above: no +// reduction-order freedom and no envelope to hide behind. That case is +// written and has NEVER RUN on this host (no `nvcc`, so it reports `no CUDA +// backend; skipping`); it is `## Owed` O6 of the row's spec and is not +// counted as coverage here. #include #include @@ -103,6 +112,61 @@ std::vector RandF32(size_t n, uint32_t seed) { return v; } +// bf16 helpers. Every published DFlash2 checkpoint stores this op's tensors in +// bf16, and bf16 is the ONLY arm on which the rounding policy is observable: on +// f32 the kernel's per-step rounding is the identity by construction, so no f32 +// case can see it. +std::vector ToBf16(const std::vector& v) { + std::vector o(v.size()); + for (size_t i = 0; i < v.size(); ++i) o[i] = vt::F32ToBF16(v[i]); + return o; +} +Tensor Bf16(std::vector& v, const std::vector& shape) { + return Contig(v.data(), DType::kBF16, Cpu(), shape); +} +// Round an f32 through bf16 and back -- one MATERIALIZATION of an intermediate. +float RB(float v) { return vt::BF16ToF32(vt::F32ToBF16(v)); } + +// UPSTREAM's reference loop again, at upstream's bf16 MATERIALIZATION points. +// Written from the upstream chain rather than from our kernel: `base + delta`, +// `coefficients * blocks` and `output += ...` each produce a tensor of the model +// dtype, so on a bf16 draft each of the three rounds, and the accumulation runs +// tap-ascending because upstream's `for tap in range(1, taps)` does. +// +// This is the whole reason the CUDA arm can be specified BIT-IDENTICAL rather +// than within an envelope, and it is not free: rounding ONCE at the end gives a +// different answer, which the hand-computed case below pins with literals. +std::vector ReferenceBf16(const std::vector& hidden, + const std::vector& delta, + const std::vector& base, int64_t batch, + int64_t block, int64_t taps, int64_t groups, + int64_t gsize, int64_t sides, int64_t side) { + const int64_t H = groups * gsize; + std::vector out(static_cast(batch * block * H), 0); + for (int64_t b = 0; b < batch; ++b) { + for (int64_t pos = 0; pos < block; ++pos) { + const int64_t row = b * block + pos; + for (int64_t g = 0; g < groups; ++g) { + for (int64_t j = 0; j < gsize; ++j) { + const int64_t c = g * gsize + j; + float acc = 0.0f; + for (int64_t tap = 0; tap < taps && tap <= pos; ++tap) { + const size_t di = + static_cast(((row * sides + side) * taps + tap) * groups + g); + const size_t bi = static_cast((side * taps + tap) * H + c); + const float k = RB(vt::BF16ToF32(base[bi]) + vt::BF16ToF32(delta[di])); + const float term = + RB(k * vt::BF16ToF32(hidden[static_cast((row - tap) * H + c)])); + acc = (tap == 0) ? term : RB(acc + term); + } + out[static_cast(row * H + c)] = vt::F32ToBF16(acc); + } + } + } + } + return out; +} + // UPSTREAM's reference loop, transcribed from // tests/v1/spec_decode/test_dflash2.py::test_grouped_conv_matches_reference @ the // PR head. `hidden` is [batch*block, H] and every request block is contiguous and @@ -258,6 +322,98 @@ TEST_CASE("dflash2-grouped-conv: base_kernel dim 0 is the SIDE, not a tap") { CHECK(got1[0] == doctest::Approx(14.0f)); } +// =========================================================================== +// bf16 — the ROUNDING POLICY, on the only arm that can see it. +// +// The cases above run in f32, where the kernel's per-step rounding is the +// identity by construction, so NOTHING above this line can tell per-step +// rounding from rounding once at the end. That distinction is not cosmetic: it +// is the reason `DFlashGroupedConvArgs` specifies the CUDA mirror as +// BIT-IDENTICAL to this CPU reference rather than within a tolerance, and it is +// what makes the op agree with upstream's bf16 chain element for element. Both +// cases below are CPU-ONLY and do not wait for a GPU. + +TEST_CASE("dflash2-grouped-conv bf16: the answer rounds PER STEP, not once at the end") { + // Hand-computed with literals, so the expectation is independent of any + // reference loop in this file. bf16 carries 8 significand bits, so above 256 + // it steps by 2 and round-to-nearest-EVEN decides every halfway case. + // + // Two blocks of 2 rows; taps 2; TWO groups of one channel each (so each + // channel gets its own delta); one side. x alternates a large row and a small + // one, base = {3, 5} per channel for both taps, delta = 2^-9 everywhere. + // + // k(ch0) = bf16(3 + 0.001953125) = 3 (the `base + delta` rounding) + // k(ch1) = bf16(5 + 0.001953125) = 5 + // + // row 0 / row 2 (pos 0, tap 1 masked by the block boundary) + // ch0: bf16(3*89) = bf16(267) = 268 (267 is halfway; 268 is the even one) + // ch1: bf16(5*53) = bf16(265) = 264 (265 is halfway; 264 is the even one) + // row 1 / row 3 (pos 1, both taps) + // ch0: bf16(bf16(3*1) + bf16(3*89)) = bf16(3 + 268) = bf16(271) = 272 + // ch1: bf16(bf16(5*1) + bf16(5*53)) = bf16(5 + 264) = bf16(269) = 268 + // + // Round ONCE at the end instead and six of these eight outputs move: + // row 0/2 ch1 -> 266, row 1/3 ch0 -> 270 and ch1 -> 270. That is the whole + // difference between the two policies, and it is why the numbers below are + // 268/264/272/268 rather than 268/266/270/270. + const float kDelta = 0.001953125f; // 2^-9, exactly representable in bf16 + std::vector x = ToBf16({89.0f, 53.0f, 1.0f, 1.0f, 89.0f, 53.0f, 1.0f, 1.0f}); + std::vector delta = ToBf16(std::vector(4 * 1 * 2 * 2, kDelta)); + std::vector base = ToBf16({3.0f, 5.0f, 3.0f, 5.0f}); + std::vector got(8, 0); + Tensor tx = Bf16(x, {4, 2}); + Tensor tc = Bf16(delta, {4, 1, 2, 2}); + Tensor tb = Bf16(base, {1, 2, 2}); + Tensor to = Bf16(got, {4, 2}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, + Args(/*block=*/2, /*taps=*/2, /*groups=*/2, /*gsize=*/1, /*side=*/0)); + const std::vector want = {268.0f, 264.0f, 272.0f, 268.0f, + 268.0f, 264.0f, 272.0f, 268.0f}; + for (size_t i = 0; i < want.size(); ++i) { + INFO("row ", i / 2, " channel ", i % 2, " got ", vt::BF16ToF32(got[i])); + CHECK(got[i] == vt::F32ToBF16(want[i])); + } +} + +TEST_CASE("dflash2-grouped-conv bf16 is BIT-EXACT against the per-step reference") { + // The same claim at shapes with real fan-in, against ReferenceBf16 — which + // rounds where UPSTREAM materializes rather than where our kernel does. Both + // published block shapes and both sides; taps 3 at block 16 so more than one + // accumulate rounding is chained. Asserted on the STORED bf16 patterns, not + // through a tolerance, because bit-identity is what the op promises. + struct Case { + int64_t batch, block, taps, groups, gsize, sides, side; + uint32_t seed; + }; + const Case cases[] = { + {2, 8, 2, 4, 2, 2, 0, 44}, {2, 8, 2, 4, 2, 2, 1, 55}, {2, 16, 3, 4, 2, 1, 0, 66}}; + for (const Case& cs : cases) { + const int64_t H = cs.groups * cs.gsize; + const int64_t T = cs.batch * cs.block; + std::vector x = ToBf16(RandF32(static_cast(T * H), cs.seed)); + std::vector delta = ToBf16( + RandF32(static_cast(T * cs.sides * cs.taps * cs.groups), cs.seed + 1)); + std::vector base = + ToBf16(RandF32(static_cast(cs.sides * cs.taps * H), cs.seed + 2)); + std::vector got(static_cast(T * H), 0); + Tensor tx = Bf16(x, {T, H}); + Tensor tc = Bf16(delta, {T, cs.sides, cs.taps, cs.groups}); + Tensor tb = Bf16(base, {cs.sides, cs.taps, H}); + Tensor to = Bf16(got, {T, H}); + Queue q = Q(); + vt::DFlashGroupedConv(q, to, tx, tc, tb, + Args(cs.block, cs.taps, cs.groups, cs.gsize, cs.side)); + const std::vector want = ReferenceBf16(x, delta, base, cs.batch, cs.block, + cs.taps, cs.groups, cs.gsize, + cs.sides, cs.side); + for (size_t i = 0; i < want.size(); ++i) { + INFO("block ", cs.block, " taps ", cs.taps, " side ", cs.side, " index ", i); + CHECK(got[i] == want[i]); + } + } +} + // =========================================================================== // CUDA parity. Unlike the attention ops, this one is elementwise with a rounding // to the tensor dtype after each materialized step, so CPU and CUDA must agree @@ -312,12 +468,6 @@ class DeviceTensor { Tensor t_; }; -std::vector ToBf16(const std::vector& v) { - std::vector o(v.size()); - for (size_t i = 0; i < v.size(); ++i) o[i] = vt::F32ToBF16(v[i]); - return o; -} - // One shape, run on CPU and CUDA in the SAME dtype, asserted BIT-EQUAL. void RunCudaParity(int64_t batch, int64_t block, int64_t taps, int64_t groups, int64_t gsize, int64_t sides, int64_t side, DType dt, uint32_t seed) { From cfbc2bd2e45f8de6e3bc2d054de03ca2f792e6da Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 20 Aug 2026 09:11:35 +0000 Subject: [PATCH 3/3] record(SPEC-DFLASH2): the kernel row's evidence counted six cases and 9410 assertions, and the rounding was in none of them (#1314) `KERNEL-DFLASH2-GROUPED-CONV` in `.agents/kernel-matrix.md` recorded the op gate as 6 cases / 9410 assertions and listed six mutations that redden it. The per-step rounding was not among them, and the wave's second fresh review proved why: every case in that file ran in f32, where the rounding is the identity by construction, so replacing the bf16 branch of the `round` lambda with `return v;` compiled clean and left both suites green. The row now records 8 cases / 9930 assertions with the two bf16 cases named, carries the rounding mutation in its set with the red it produces (2 cases and 225 assertions failed, `Status: FAILURE!`), counts THREE gate repairs from this row's mutation passes rather than two, and states beside the CUDA entry that the policy is pinned on CPU while CPU == CUDA bit-identity is pinned on neither side. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/kernel-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index 7688cab63..57fb2e648 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -140,7 +140,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:509](../include/vllm/v1/attention/backend.h#L509), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | | `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:1843](../src/vt/cpu/cpu_ops.cpp#L1843) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:1300](../src/vt/cuda/cuda_ops.cu#L1300) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | | `KERNEL-ATTN-DFLASH-PAGED-BLOCK` | **DFlash PAGED in-block attention — the CAPTURE-SAFE form of `KERNEL-ATTN-DFLASH-BLOCK`** (SPEC-DFLASH D12 Part B, the CUDA-graph draft-attention primitive). The (1+k) block queries attend over `[PAGED context ; their own (1+k) block]`: the growing context enters as DATA (paged K/V cache `[pages,block_size,Hkv,D]` + per-request `seq_lens` + `block_table`, mirroring `PagedAttentionKernel`) instead of a variable-size materialized combined buffer, so the launch grid is STATIC over the fixed `Nq=(1+k)*num_reqs` rows and EVERY metadata input is a persistent DEVICE tensor read in place — NO `cudaMallocAsync`/`cudaMemcpyAsync` of a function-local host `cu_seqlens` (the [[cudagraph-capture-bakes-stack-addresses]] UAF class the eager `LaunchDFlashBlockAttention` had). Same f32 online softmax + D2 in-block mask over the COMBINED index; bit-identical to `DFlashBlockAttention` over the materialized `[context;block]` buffer | vLLM full CG `dflash/cudagraph.py` + `speculator.py:411-458` + `precompute_and_store_context_kv` (`qwen3_dflash.py:548-619`) @ `555967922`; paged read mirrors our `PagedAttentionKernel` [cuda_paged_attn.cu:184](../src/vt/cuda/cuda_paged_attn.cu#L184) | `OpId::kDFlashPagedBlockAttention` + `DFlashPagedBlockAttentionArgs` + decl [ops.h](../include/vt/ops.h) + wrapper/validation [ops.cpp](../src/vt/ops.cpp); CPU REFERENCE `DFlashPagedBlockAttentionKernel` [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp); CUDA `DFlashPagedBlockAttentionKernelCuda` [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1452) (static grid, persistent device metadata) + D14 WARP variant [DFlashPagedBlockAttentionWarpKernel](../src/vt/cuda/cuda_ops.cu#L1433) | **GPU GATE GREEN on dgx (2026-07-27, GB10 sm_121a):** CUDA `-Werror` clean (0 warnings); [test_ops_dflash_paged_block_attn.cpp](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) cross-checks CPU-paged == materialized `DFlashBlockAttention` across 6 corners (non-causal, causal-SWA, block isolation, GQA, multi-page, zero-context) + CUDA==CPU (f32+bf16) = **795648/795648 assertions**; `compute-sanitizer --tool memcheck` **0 errors**. **D13 (2026-07-27, `CLAIM-DFLASH-D13`): WIRED INTO PRODUCTION** — the single-request DFlash draft forward (`ForwardPagedBody`, `qwen3_dflash.cpp`) now runs the (1+k) block through this kernel reading a fixed-capacity paged `DflashDeviceKVStore`, and the whole draft step is captured into a per-request CUDA graph + replayed (the growing context enters only via the in-place `seq_lens`). Capture-correctness PROVEN: `test_qwen27_dflash_spec_decode` 27/27 with the graph BIT-IDENTICAL to eager (same tokens + acceptance 19/39/29/25); c1 throughput NEAR-PARITY with vLLM-DFlash-ON (ours 0.978×, ~2% below the tight 3-rep band; gap closed 0.917×→0.978× via the paged read, the CG is perf-neutral) — the kernel is landed + wired + gated; STAYS `ACTIVE` with the engine feature (the ~2% ≥vLLM residual is per-step compute for an nsys). **D14 (2026-07-27, `CLAIM-DFLASH-D14`): the residual WAS this kernel → WARP-scoped variant added → SPEED GATE MET, `DONE`.** An nsys (`--cuda-graph-trace=node`) attributed the D13 ~2% residual to THIS kernel: `DFlashPagedBlockAttentionKernel` = 242.9 ms = 1.8% of the graphed step's GPU time, median ~460 us/call (grid `(nq=17,hq=32)` × kBlock=256 threads looping SERIALLY over C~500-640 keys with a 256-wide shared-mem tree reduction + 2 `__syncthreads` PER key — the latency/sync storm the ViT tower fixed with `AttentionDenseFast`), vs vLLM's fused flash draft-attn ~0.15%. Added `DFlashPagedBlockAttentionWarpKernel` ([cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)): ONE WARP per (block-query,head), `__shfl_xor` butterfly head_dim reduction, register accumulator, NO `__syncthreads`; SAME paged/block combined-index read + causal/SWA mask + GQA (copied verbatim from the block kernel), mirroring the shipped `AttentionWarpKernel`. Default ON; `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel. Draft attn **242.9 → 77.9 ms (3.1×)**; our-ON c1 **28.60 → 29.32 tok/s**; FINAL 3-rep A/B our-ON 29.32 ≥ vLLM-ON 29.240 (non-overlapping bands, 1.003×) ⇒ **≥vLLM MET**. Not bit-identical to the block kernel but same f32-online-softmax math within envelope; CUDA==CPU `test_ops_dflash_paged_block_attn` **795648/795648** (f32 1e-4/bf16 3e-2) + **compute-sanitizer 0**; e2e 27/27 graph==eager, acceptance 19/39/29/25 unchanged (1629 accepted identical warp-vs-block); SACRED 235/235 + MTP 9/9 inert; `-Werror` clean; closure [ledger](parity-ledger.md#L738) | [DFlash spec](specs/dflash-spec-decode.md) §0 D12/D13/D14 | `DONE` | `489a7544` | -| `KERNEL-DFLASH2-GROUPED-CONV` | **DFlash2 grouped dynamic depthwise convolution — the project's FIRST dynamic (input-conditioned) convolution kernel** (SPEC-DFLASH2 W2, #1314). `out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, with tap `t` contributing only where `(i mod block_size) >= t`, `g(c) = c / group_size`, and `block_size` the QUERY block `1 + k`. Three things separate it from the shipped `KERNEL-DEPTHWISE-CONV1D`: the kernel is DYNAMIC (a per-position `delta` projected from the sublayer input, added to a static per-channel `base`), it is GROUPED (one delta per group of channels, one base per channel), and its taps are ZEROED ACROSS THE BLOCK BOUNDARY rather than across the sequence — which is what lets a proposal position see the ones before it without another backbone pass. `base_kernel` dim 0 is the prepare/finish SIDE and not a tap; on the published 27B draft both axes are 2, so nothing but the port note and the shape assertion separates a correct load from a transposed one. Every intermediate rounds to the tensor dtype, mirroring upstream's bf16 chain, so the op is elementwise with NO reduction-order freedom and the CUDA arm is specified BIT-IDENTICAL to CPU rather than within an envelope | **BEYOND-PIN** — `vllm/model_executor/models/qwen3_dflash2.py` (`_grouped_conv`, `DFlashGroupedConv`, `DFlash2Qwen3DecoderLayer.forward`) @ [vllm-project/vllm#52816](https://github.com/vllm-project/vllm/pull/52816) head `19c9351904df4c63042671bc67a866ca48dc7d6f`; the parity pin `555967922` does not carry the architecture and this row does NOT advance it | `OpId::kDFlashGroupedConv` + `DFlashGroupedConvArgs` + decl/wrapper `include/vt/ops.h::DFlashGroupedConv` and `src/vt/ops.cpp::DFlashGroupedConv`; CPU REFERENCE `src/vt/cpu/cpu_ops.cpp::DFlashGroupedConvKernel` (the authoritative impl); CUDA mirror `src/vt/cuda/cuda_ops.cu::DFlashGroupedConvKernelCuda` (one thread per (row, channel); `__fadd_rn`/`__fmul_rn` forbid the FMA contraction the CPU build pins off). Consumed by the draft through `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvPrepare` and `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvFinish`, called from all THREE layer bodies, with the uniform-block precondition in `src/vllm/model_executor/models/qwen3_dflash.cpp::CheckDflashConvBatch`; weights loaded by `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash` | **CPU GATE GREEN 2026-08-19** ([test_ops_dflash2_grouped_conv.cpp](../tests/vt/test_ops_dflash2_grouped_conv.cpp)) 6 cases / **9410 assertions**, `Status: SUCCESS!`, exit 0 — upstream's own sequential reference loop at block 5 (the `% block` arm), 8 and 16 (the two PUBLISHED checkpoints, `z-lab/Qwen3.8-27B-DFlash2` and `z-lab/Muse-Glimmer-30B-DFlash2`; upstream's parametrize covers 5 and 8 only), both published taps/group shapes on both sides, plus hand-computed corners for the block boundary, the group map and the side. MODEL GATE GREEN ([test_qwen3_dflash2_draft.cpp](../tests/vllm/models/test_qwen3_dflash2_draft.cpp)) 16 cases / 108 assertions, `Status: SUCCESS!`, exit 0 — weights read off a REAL on-disk safetensors shard by the production loader, an IDENTITY conv proven BIT-IDENTICAL to no conv, and each conv driven ALONE through each of the three layer bodies. MUTATION-PROVEN 2026-08-19, each restored byte-for-byte and verified by sha256: deleting the call sites in `ForwardBlockLogits` (5 cases / 9 assertions red), in `ForwardWithCtxKVDev` (1/1) and in `ForwardPagedBody` (1/1); forcing `args.side` to 0 (op 2 cases/4353 assertions red, model 1/1); dropping the block mask (3/449); the wrong group map (3/7436); and dropping the uniform-block guard (1/1). TWO gate repairs came out of that pass and are recorded rather than hidden: activating both convs at once could not see one missing call site, and the first side probe could not see a forced side. **CUDA UNVERIFIED and OWED** — the kernel and its registration are written and the CUDA==CPU bit-identity case exists over six shapes, but the authoring host has no `nvcc`, so it has NEVER COMPILED and the case reports `no CUDA backend; skipping`. Spec `## Owed` O6, owed to the operator's GPU lease | [DFlash2 spec](specs/dflash2-spec-decode.md) W2, [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `ACTIVE` | `CLAIM-SPEC-DFLASH2-W2` | +| `KERNEL-DFLASH2-GROUPED-CONV` | **DFlash2 grouped dynamic depthwise convolution — the project's FIRST dynamic (input-conditioned) convolution kernel** (SPEC-DFLASH2 W2, #1314). `out[i,c] = sum_t (base[side,t,c] + delta[i,side,t,g(c)]) * x[i-t,c]`, with tap `t` contributing only where `(i mod block_size) >= t`, `g(c) = c / group_size`, and `block_size` the QUERY block `1 + k`. Three things separate it from the shipped `KERNEL-DEPTHWISE-CONV1D`: the kernel is DYNAMIC (a per-position `delta` projected from the sublayer input, added to a static per-channel `base`), it is GROUPED (one delta per group of channels, one base per channel), and its taps are ZEROED ACROSS THE BLOCK BOUNDARY rather than across the sequence — which is what lets a proposal position see the ones before it without another backbone pass. `base_kernel` dim 0 is the prepare/finish SIDE and not a tap; on the published 27B draft both axes are 2, so nothing but the port note and the shape assertion separates a correct load from a transposed one. Every intermediate rounds to the tensor dtype, mirroring upstream's bf16 chain, so the op is elementwise with NO reduction-order freedom and the CUDA arm is specified BIT-IDENTICAL to CPU rather than within an envelope | **BEYOND-PIN** — `vllm/model_executor/models/qwen3_dflash2.py` (`_grouped_conv`, `DFlashGroupedConv`, `DFlash2Qwen3DecoderLayer.forward`) @ [vllm-project/vllm#52816](https://github.com/vllm-project/vllm/pull/52816) head `19c9351904df4c63042671bc67a866ca48dc7d6f`; the parity pin `555967922` does not carry the architecture and this row does NOT advance it | `OpId::kDFlashGroupedConv` + `DFlashGroupedConvArgs` + decl/wrapper `include/vt/ops.h::DFlashGroupedConv` and `src/vt/ops.cpp::DFlashGroupedConv`; CPU REFERENCE `src/vt/cpu/cpu_ops.cpp::DFlashGroupedConvKernel` (the authoritative impl); CUDA mirror `src/vt/cuda/cuda_ops.cu::DFlashGroupedConvKernelCuda` (one thread per (row, channel); `__fadd_rn`/`__fmul_rn` forbid the FMA contraction the CPU build pins off). Consumed by the draft through `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvPrepare` and `src/vllm/model_executor/models/qwen3_dflash.cpp::DflashConvFinish`, called from all THREE layer bodies, with the uniform-block precondition in `src/vllm/model_executor/models/qwen3_dflash.cpp::CheckDflashConvBatch`; weights loaded by `src/vllm/model_executor/models/qwen3_dflash_weights.cpp::LoadQwen3DFlash` | **CPU GATE GREEN 2026-08-19** ([test_ops_dflash2_grouped_conv.cpp](../tests/vt/test_ops_dflash2_grouped_conv.cpp)) 8 cases / **9930 assertions**, `Status: SUCCESS!`, exit 0 (was 6 / 9410 on 2026-08-19, before the bf16 rounding cases below) — upstream's own sequential reference loop at block 5 (the `% block` arm), 8 and 16 (the two PUBLISHED checkpoints, `z-lab/Qwen3.8-27B-DFlash2` and `z-lab/Muse-Glimmer-30B-DFlash2`; upstream's parametrize covers 5 and 8 only), both published taps/group shapes on both sides, plus hand-computed corners for the block boundary, the group map and the side. MODEL GATE GREEN ([test_qwen3_dflash2_draft.cpp](../tests/vllm/models/test_qwen3_dflash2_draft.cpp)) 16 cases / 108 assertions, `Status: SUCCESS!`, exit 0 — weights read off a REAL on-disk safetensors shard by the production loader, an IDENTITY conv proven BIT-IDENTICAL to no conv, and each conv driven ALONE through each of the three layer bodies. MUTATION-PROVEN 2026-08-19, each restored byte-for-byte and verified by sha256: deleting the call sites in `ForwardBlockLogits` (5 cases / 9 assertions red), in `ForwardWithCtxKVDev` (1/1) and in `ForwardPagedBody` (1/1); forcing `args.side` to 0 (op 2 cases/4353 assertions red, model 1/1); dropping the block mask (3/449); the wrong group map (3/7436); and dropping the uniform-block guard (1/1). **The PER-STEP ROUNDING was added to that set on 2026-08-20**, after the wave's second fresh review proved it had no executing assertion: replacing the bf16 branch of the `round` lambda in `src/vt/cpu/cpu_ops.cpp::DFlashGroupedConvKernel` with `return v;` compiled clean and left BOTH suites fully green, because every case in the op file ran in f32 where that rounding is the IDENTITY, and the model suite asserts only RELATIONALLY between two runs of the same kernel. Two CPU-only bf16 cases now pin it — one hand-computed against literals that differ from the round-once-at-the-end answer in six of eight outputs, one bit-exact at three shapes against a reference that rounds where UPSTREAM materializes — and under the same mutation the file is 8 cases / 2 failed, 9930 assertions / 225 failed, `Status: FAILURE!`. THREE gate repairs have now come out of this row's mutation passes and are recorded rather than hidden: activating both convs at once could not see one missing call site, the first side probe could not see a forced side, and no case at all could see the rounding policy. **CUDA UNVERIFIED and OWED** — the kernel and its registration are written and the CUDA==CPU bit-identity case exists over six shapes, but the authoring host has no `nvcc`, so it has NEVER COMPILED and the case reports `no CUDA backend; skipping`. The per-step rounding policy is pinned on CPU; CPU == CUDA bit-identity is pinned on NEITHER side. Spec `## Owed` O6, owed to the operator's GPU lease | [DFlash2 spec](specs/dflash2-spec-decode.md) W2, [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `ACTIVE` | `CLAIM-SPEC-DFLASH2-W2` | | `KERNEL-ATTN-DSA-SPARSE-INDEX` | **DeepSeek-V4 DSA "Lightning Indexer" sparse-attention SELECTION — the project's FIRST sparse candidate-selection primitive** (DeepSeek-V4-Flash W3). Two ops: (1) the weighted-MQA INDEXER LOGIT `logit[t,s] = Σ_h w[t,h]·ReLU(q[t,h]·k[s])` over the causal candidate window (the per-head **ReLU** is load-bearing — it is what makes the indexer a learned sparse SELECTOR, not a plain attention score), where `w[t,h] = weights_proj[t,h]·index_head_dim^-0.5·index_n_heads^-0.5`; and (2) the per-row **causal top-k** that keeps the `index_topk=512` highest-logit keys (short-context: every candidate, ascending; else top-k with -1 padding). Distinct from every dense/paged/MLA family, which score ALL keys — this one PICKS a sparse key subset the downstream MLA then attends over. W3 also lands the two 512-wide-MLA OUTPUT seams V2/V3 lack (per-head attention-**sink** softmax + **grouped output-LoRA** `wo_a` bmm→`wo_b`) as portable host references beside it | MQA logit `vllm/v1/attention/ops/triton_fp8_mqa_logits.py:120-156` (dot→×kv_scale→ReLU→×weights→Σheads); weight fold `vllm/model_executor/layers/sparse_attn_indexer.py:203-207`; top-k `sparse_attn_indexer.py:488-497` + short-context `vllm/models/deepseek_v4/attention.py:70-86,:813-831`; sinks `deepseek_v4/nvidia/flashinfer_sparse.py:777,:896`; grouped output-LoRA `deepseek_v4/nvidia/ops/o_proj.py:58-73` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_dsa.cpp](../src/vllm/model_executor/models/deepseek_v4_dsa.cpp) + [deepseek_v4_dsa.h](../include/vllm/model_executor/models/deepseek_v4_dsa.h): `DsaIndexerWeightFold` / `DsaIndexerLogits` / `DsaTopkSelect` / `SoftmaxWithSink` / `GroupedOutputLora` | **CPU UNIT GATE GREEN (2026-07-28, `-Wall -Werror -Wextra` 0-warn):** [test_deepseek_v4_dsa.cpp](../tests/vllm/models/test_deepseek_v4_dsa.cpp) **13/13 cases · 38 assertions** — hand-derived literal cases (the ReLU clip, the weight fold, short-context all-select, full top-k, tie→smaller-index, causal-window offset, sink probability mass, sink numerical stability, grouped-LoRA) + from-first-principles double-precision references on randomized shapes (indexer logits + grouped output-LoRA rel-L2 < 1e-6). Full-model gate is multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W3 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W3` | | `KERNEL-ATTN-DSA-COMPRESSOR` | **DeepSeek-V4 DSA COMPRESSOR + fp8_ds_mla KV-cache state — the second half of the sparse-attention stack** (DeepSeek-V4-Flash W4). Where `KERNEL-ATTN-DSA-SPARSE-INDEX` SELECTS keys, this POOLS + QUANTIZES them into the compressed latent the MLA reads and defines how it is cached across steps. Three ops: **(1)** the softmax-weighted window POOL — at a compress boundary the compressor gathers `(1+overlap)·compress_ratio` KV-state rows and computes, PER head-dim column, `softmax(score, dim=0)·kv` (each channel pools the window with its OWN weights — the load-bearing nuance), then RMSNorm; **(2)** the fused save-time APE add `score_state = score + ape[position % compress_ratio]`; **(3)** the **fp8_ds_mla** KV-cache STATE layout — the 512-wide latent split into a 448-wide NoPE part quantized to FP8 e4m3 with per-64 **UE8M0** power-of-two block scales (exponent `= ceil(log2(absmax/448))`, byte `= exp+127`) and a 64-wide RoPE part stored bf16, at a **576-byte** token stride with a padded **7+1** scale region — plus the dequant READ (`nope = e4m3·2^(byte-127)`, `rope = bf16`) | pool+RMSNorm `vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py:198-218`; save-time APE `common/ops/save_partial_states.py:92-101`; fp8_ds_mla store `fused_compress_quant_cache.py:220-297`; layout `deepseek_v4/compressor.py:307-309`; dequant READ cross-checked to SGLang `v0.5.15` `dsv4/dequant_k_cache.py:12-18,:122-136` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_compressor.cpp](../src/vllm/model_executor/models/deepseek_v4_compressor.cpp) + [deepseek_v4_compressor.h](../include/vllm/model_executor/models/deepseek_v4_compressor.h): `CompressorSaveScoreApe` / `CompressorPoolNorm` / `MakeFp8DsMlaLayout` / `Fp8DsMlaEncodeToken` / `Fp8DsMlaDecodeToken` | **CPU UNIT GATE GREEN (2026-07-29, Debug full-library build, 0-warn on the new TUs):** [test_deepseek_v4_compressor.cpp](../tests/vllm/models/test_deepseek_v4_compressor.cpp) **12/12 cases · 164 assertions** — hand-derived literal cases (APE modulo wrap; per-column softmax pool proven load-bearing via the column-ratio-survives-RMSNorm case; window masking; V4 layout 448/64/576/7+1; all-ones→UE8M0 byte 119 exact round-trip; value-3→byte 120; bf16 rope verbatim) + from-first-principles double-precision references (pool+norm rel-L2 < 1e-6; independent UE8M0 scale-byte recompute; encode→decode round-trip < 0.05 fp8 granularity). RED-first PROVEN: perturbing the scale bias `+127→+126` fails 4 cases / 135 assertions; revert restores 12/12. Honest gate form: hand-case + structural review vs vLLM+SGLang `file:line` (fixed-config 167B not constructible at a tiny shape ⇒ NOT a dumped-oracle rel-L2). Full-model gate multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + the fused device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W4 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W4` | | `KERNEL-MHC-SINKHORN` | **DeepSeek-V4 Manifold/Markov Hyper-Connections (MHC) — the Sinkhorn-normalized hc_mult-stream residual manifold** (DeepSeek-V4-Flash W5, the hardest V4 brick). V4 replaces the plain `residual + RMSNorm` stream with a `[tokens, hc_mult=4, hidden]` MANIFOLD of parallel residual streams, mixed at every attn/ffn boundary by a **doubly-stochastic** matrix and collapsed by a learned head. Four ops: **(1)** the **`hc_sinkhorn_iters=20` Sinkhorn** normalization of the hc_mult×hc_mult mixing matrix — a row-softmax seed (`+eps`), a col-norm, then `(iters-1)×[row-norm, col-norm]` toward a doubly-stochastic matrix (the AXIS ALTERNATION and the ITERATION COUNT are load-bearing at non-converged counts — RED-first proven); **(2)** the mHC **pre** mix — flatten the streams, project through `hc_*_fn` with a FOLDED weight-free RMSNorm `rsqrt(sqrsum/(hc·H)+rms_eps)`, split into pre/post/comb gates (`pre=σ+hc_eps`, `post=σ·hc_post_alpha(2.0)`, `comb=Sinkhorn`), collapse to the single `layer_input`, and optionally FOLD the model's attn_norm/ffn_norm RMSNorm; **(3)** the mHC **post** mix — fold the block output back into the manifold via the comb matrix (`Σ_i comb[i,j]·res[i,h]`) + the post gate; **(4)** the **hc_head** collapse — weight-free RMSNorm → `hc_head_fn` → sigmoid gate → weighted stream sum → one hidden vector. **EAGER-REF FINDING: corrects the W0 "ZERO eager reference upstream" premise** — the pinned vLLM DOES ship an eager PyTorch reference (`mhc/torch.py` `mhc_pre_torch`/`mhc_post_torch`, `triton.py` head collapse); four upstream impls (torch.py, tilelang_kernels.py `_sinkhorn_fwd`, tilelang.py, SGLang mhc.py) agree byte-for-byte on the Sinkhorn | mHC pre/post + Sinkhorn `vllm/model_executor/kernels/mhc/torch.py:56-106` (byte-identical `tilelang_kernels.py:126-153` `_sinkhorn_fwd`, `tilelang.py` `mhc_pre_big_fuse_with_norm`); head collapse `triton.py:108-140` + `tilelang.py:720-748`; constants `hc_post_alpha=2.0`/`hc_pre_eps=hc_sinkhorn_eps=hc_eps` `vllm/models/deepseek_v4/nvidia/model.py:818-821,:886-894,:1023-1041`; cross-checked SGLang `v0.5.15` `python/sglang/srt/layers/mhc.py:110-126` @ `555967922` | Portable host reference (device kernel + `DeepseekV4Model::Forward` assembly are W7 residuals) [deepseek_v4_mhc.cpp](../src/vllm/model_executor/models/deepseek_v4_mhc.cpp) + [deepseek_v4_mhc.h](../include/vllm/model_executor/models/deepseek_v4_mhc.h): `MhcSinkhorn` / `MhcPre` / `MhcPost` / `HcHeadCollapse` | **CPU UNIT GATE GREEN (2026-07-29, Debug full-library build, `-Wall -Werror -Wextra` 0-warn on the new TUs):** [test_deepseek_v4_mhc.cpp](../tests/vllm/models/test_deepseek_v4_mhc.cpp) **14/14 cases · 125 assertions** — hand-derived literal cases (all-zero Sinkhorn → uniform doubly-stochastic 1/hc; symmetric-2×2 fixed point `[[.75,.25],[.25,.75]]`; iteration-count load-bearing; MhcPre fn=0 gate midpoints; RMSNorm fold `[1,3]→[1,3]/√5`; MhcPost identity-comb + post-add; mix sums over the first comb index; hc_head fn=0 stream mean) + from-first-principles DOUBLE-PRECISION references (Sinkhorn/MhcPre/MhcPost/HcHead f32==f64 rel-L2 < 1e-5..1e-4; doubly-stochastic convergence to row/col sums=1). **RED-first PROVEN both levers:** perturb the Sinkhorn iteration count (`iters-1→iters-2`) fails 1 case/9 assertions AND swap a normalization axis fails 2 cases/12 assertions (caught by a dedicated SMALL-iteration-count gate, since at 20 iters the Sinkhorn has converged and ±1 is within tolerance); revert restores 14/14·125. Honest gate form: DERIVED-eager-reference + hand-case + structural review vs vLLM+SGLang `file:line` (fixed-config 167B not constructible at a tiny shape ⇒ NOT a dumped-oracle rel-L2). OPEN QUESTION: end-to-end bf16 residual/layer_input rounding between steps is a W7 device concern, not folded into these f32/f64 refs. Full-model gate multi-Spark-blocked (156.7 GiB); sqrtsoftplus/hash MoE (W6) + device kernel + forward assembly (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W5 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W5` |