feat(vulkan): both GDN recurrences go native; prefill amortises the state tile

.agents/specs/vulkan-full-support.md

@@ -327,6 +327,53 @@ the umbrella, not a substitute for them.
 | **VK-H** | **Attention variants + samplers** (16 ops) | B (samplers), G (attn variants) | **83/83 — closes the op surface** |
 | **VK-I** | **AMD/RDNA (or Arc) bring-up** | hardware acquisition | The staging path for non-host-visible memory, and the gate re-run where Vulkan actually matters |
 
+### 6.0 `VK-G` partial: the two GATED-DELTA RECURRENCES landed — 2026-08-08
+
+`row/BACKEND-VULKAN-GDN-CORE`. `kGdnPrefill` and `kGdnDecode` are native; the
+Vulkan module count goes 22 -> 24 and the GDN family 6 -> 8. `kCausalConv1dFwd`,
+`kRopeCosSinCache` and `kAttnQkNormRopeGate` stay on the reference tier.
+
+**What was ported, and from where.** Per-step arithmetic 1:1 from
+`src/vt/cpu/cpu_ops.cpp:1280-1311` `GdnHeadTokenStep`; dispatch shape and state
+handling from our own `src/vt/cuda/cuda_gdn.cu:2417-2503` `GdnDecodeFusedKernel`
+— one workgroup per (sequence, value-head, BV-value-tile), the `[BV,Dk]` state
+slice staged into shared memory by a coalesced load and written back once, `NW`
+lanes per value row splitting the `Dk` contraction. `BV=16`, `NW=8` are what
+Vulkan's GUARANTEED 16 KB of shared memory allows at `VT_TG=128`; `BV=32` would
+need 18 KB. Both shaders share one step body, `vt_gdn_recurrence.glsl`.
+
+**THE ONE STRUCTURAL ADDITION over the CUDA kernel, and it is the prefill lever.**
+CUDA's fused kernel is a DECODE kernel, so its coalesced load/store pair brackets
+a single step. Prefill runs the SAME staged tile through the whole token range,
+so a 512-token prompt touches the `[Dv,Dk]` state twice instead of ~2048 times.
+That is legal because the VALUE ROW is an independent axis — row `vi` of `S` only
+ever depends on itself, `k`, `q`, `decay`, `beta` and `v[vi]` — which CUDA already
+exploits as its `grid.x` value tiling; only the sequence position is sequential,
+and it stays sequential inside the workgroup.
+
+**Measured (llvmpipe, correctness only — no speed claim is made or owed here).**
+NMSE vs the CPU oracle in the same binary: prefill out `1.47e-14`, prefill
+carried state `6.43e-15`, bf16 arm `0`; decode (indexed cache) out `1.64e-14` and
+cache `3.31e-15`, decode (compact state) out `1.75e-14`. `test_vulkan_backend`
+25/25 cases, 1020/1020 assertions. `test_opt_paged_engine` with
+`VLLM_CPP_DEVICE=vulkan` still 6/6 token-exact (96/96), 0 declines.
+
+**NOT measured: any speed number.** Local Vulkan is llvmpipe. The 27B
+prefill/decode re-run on GB10, and the reference-tier count that goes with it,
+are OWED and are the only thing that can turn the structure above into a result.
+
+**A LATENT SEAM DEFECT THIS ROW HAD TO FIX.** `GetOpFallback` threw for any
+native kernel that declined per-call, because the portable reference tier
+installs only on a GetOp MISS and an op with a native kernel never misses. The
+pre-existing `vt_paged_attn` fp8-KV decline had the same hole and had simply
+never been exercised. `src/vt/op_provider.cpp` now installs the tier on that path
+and drains the backend before handing a HOST kernel device memory.
+
+**NOT taken, deliberately.** Rewriting the two passes as
+`o = decay*(S·q) + v'*(k·q)` would read the state once instead of twice, but it
+is an algebraic reassociation and would move the numbers off the CPU reference's
+rounding for a saving that cannot be measured on llvmpipe. Recorded, not shipped.
+
 ### 6.1 `VK-A1` landed — 2026-08-06 (`CLAIM-VULKAN-FULL-1`)
 
 **DECISION: keep the committed-SPIR-V route; make SPECIALIZATION CONSTANTS the

docs/BENCHMARKS.md

@@ -348,7 +348,7 @@ built on it rather than keeping the flattering one.
 | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config |
 | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box |
 | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs |
-| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 22 NATIVE kernels (+6 GDN glue, CPU-oracle gated, no speed); 65 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
+| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 24 NATIVE (+8 GDN, BOTH recurrences; oracle-gated, no speed); 63 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
 | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist |

docs/STATUS.md

@@ -410,8 +410,8 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact
 ## Not supported yet
 
 LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU,
-Vulkan (opt-125m exact, GEMV 1.8x; 22 native, +6 GDN glue gated,
-recurrences host-tier; qwen3_5 #125 fixed, VERIFIED on 27B; CUDA build repaired
+Vulkan (opt-125m exact, GEMV 1.8x; 24 native, +8 GDN incl. BOTH
+recurrences, oracle-gated, no speed; qwen3_5 #125 VERIFIED on 27B; CUDA build repaired
 [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green
 on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in —
 **blind-written, unverified** — M2 unblocks on verification; gfx1201 hipBLAS +

src/vt/op_provider.cpp

@@ -374,11 +374,32 @@ void* GetOpFallback(OpId op, DeviceType device, const char* declining_provider)
                                  "' is not registered for this op/device");
   const ProviderCaps caps = GetDeviceProviderCaps(device);
   const OpProvider* next = Choose(slot, caps, floor);
+  // NOTHING BELOW: install the portable reference tier and re-select. Resolve()
+  // installs it only on a GetOp MISS, so an op whose device HAS a native kernel
+  // never gets one — and a native kernel that declines per-call then had nothing
+  // to fall back to and threw. That is the wrong answer twice over: the tier is
+  // the whole reason a declining kernel is preferable to a throwing one
+  // (op_provider.h § DECLINE-AND-FALL-BACK), and before the native kernel existed
+  // this very shape resolved to the tier and worked. `floor` stays valid across
+  // the install because providers[] is a fixed array appended in place.
+  if (next == nullptr && MaybeInstallReferenceTier(op, device)) {
+    next = Choose(slot, caps, floor);
+  }
   slot.declines.fetch_add(1, std::memory_order_relaxed);
   VT_CHECK(next != nullptr,
            std::string("provider '") + declining_provider + "' declined op " +
                std::to_string(static_cast<int>(op)) + " on device type " +
                std::to_string(static_cast<int>(device)) + " and no provider is below it");
+  // The reference tier is a HOST kernel about to read and write DEVICE memory,
+  // exactly as in GetOp above — and GetOp's drain is keyed on the SELECTED
+  // provider, which on this path is the declining NATIVE one, so it never fires.
+  // Without this a backend that defers submission (Vulkan batches command
+  // buffers) hands the host kernel bytes the device has not written yet, and it
+  // does so SILENTLY.
+  if (std::strcmp(next->name, kReferenceProviderName) == 0) {
+    Backend* b = TryGetBackend(device);
+    if (b != nullptr) b->FlushPending();
+  }
   return next->fn;
 }
 

src/vt/vulkan/shaders/vt_gdn_decode.comp

@@ -0,0 +1,96 @@
+#version 450
+// vt::GdnDecode (kGdnDecode) — ONE gated-delta step per batch token. Ported 1:1
+// from src/vt/cpu/cpu_ops.cpp:1368-1396 GdnDecodeKernel for the semantics, and
+// from src/vt/cuda/cuda_gdn.cu:2417-2503 GdnDecodeFusedKernel for the shape; the
+// step body and the shared-memory state tiling live in vt_gdn_recurrence.glsl,
+// which cites both.
+//
+// This is the SAME shader as vt_gdn_prefill with the token loop collapsed to one
+// iteration, which is not a coincidence: the CPU kernels differ only in their
+// parallel axis (batch here, (sequence, value-head) there) and in the state-row
+// indirection below. The recurrence carries nothing across tokens at T==1, so the
+// tile's coalesced load/store pair brackets a single step — exactly CUDA's fused
+// decode kernel.
+//
+// state_idx (fla ssm_state_indices, cuda_gdn.cu:2432-2443): when present the
+// block reads and writes persistent-cache row state_idx[i_n] rather than row
+// i_n. A NEGATIVE index is fla's NULL block: zero the output row and skip. The
+// UPPER bound is guarded here too, which CUDA does not do, for the reason
+// vt_gdn_state_gather.comp states: the CPU's host-side range VT_CHECK would need
+// a host read of device memory, which on this backend means draining the dispatch
+// batch on every call, so the device analogue is a guard that writes nothing
+// wild. i_n is fixed by the workgroup id, so the branch is UNIFORM and taking it
+// before any barrier is safe.
+//
+// EXPECT NO LARGE DECODE WIN FROM THIS. Batch-1 decode of a 27B model is
+// weight-bandwidth bound, not recurrence bound; this kernel exists so the op
+// leaves the host reference tier and so the whole GDN block runs on one device,
+// not because the recurrence was the decode cost.
+#extension GL_GOOGLE_include_directive : require
+#include "vt_common.glsl"
+
+layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in;
+
+// IDENTICAL BINDINGS TO vt_gdn_prefill, and that is a requirement rather than a
+// tidiness: GLSL functions cannot take buffer blocks as parameters, so the shared
+// step in vt_gdn_recurrence.glsl names these instances directly.
+layout(binding = 0) readonly buffer Qb32 { uint v[]; } Q32;   // q [T,Hk,Dk]
+layout(binding = 1) readonly buffer Qb16 { uint16_t v[]; } Q16;
+layout(binding = 2) readonly buffer Kb32 { uint v[]; } K32;   // k [T,Hk,Dk]
+layout(binding = 3) readonly buffer Kb16 { uint16_t v[]; } K16;
+layout(binding = 4) readonly buffer Vb32 { uint v[]; } V32;   // v [T,Hv,Dv]
+layout(binding = 5) readonly buffer Vb16 { uint16_t v[]; } V16;
+layout(binding = 6) buffer Ob32 { uint v[]; } O32;            // out [T,Hv,Dv]
+layout(binding = 7) buffer Ob16 { uint16_t v[]; } O16;
+layout(binding = 8) readonly buffer Gb { uint v[]; } G;       // g [T,Hv], f32
+layout(binding = 9) readonly buffer Eb { uint v[]; } BETA;    // beta [T,Hv], f32
+layout(binding = 10) buffer Sb { uint v[]; } S;               // state [N,Hv,Dv,Dk], f32
+// state_idx [T], i32. Aliases the state buffer when the caller passed none
+// (p.has_idx == 0) — a descriptor a shader statically uses must be valid even on
+// the path that never reads it, the same arrangement vt_gdn_state_gather uses for
+// its optional has_initial_state.
+layout(binding = 11) readonly buffer Mb { uint v[]; } IDX;
+
+layout(constant_id = 0) const uint VT_PC_QKV_DT = VT_DT_F32;
+layout(constant_id = 1) const uint VT_PC_OUT_DT = VT_DT_F32;
+
+layout(push_constant) uniform Params {
+  uint hk, dk, hv, dv;
+  uint nv;            // value tiles per (token, head): ceil(Dv / VT_GDN_BV)
+  uint ratio;         // Hv / Hk, the GQA broadcast
+  uint has_idx;       // 1 => the state row is IDX[i_n], else i_n
+  uint n_state_rows;  // state.shape[0], for the bounds guard
+  uint q_off, k_off, v_off, out_off, g_off, beta_off, state_off, meta_off;
+  float scale;
+} p;
+
+#include "vt_gdn_recurrence.glsl"
+
+void main() {
+  uint wg = gl_WorkGroupID.x;
+  uint i_v = wg % p.nv;
+  uint nh = wg / p.nv;
+  uint h_v = nh % p.hv;
+  uint i_n = nh / p.hv;  // decode token == sequence
+  uint h_k = h_v / p.ratio;
+  uint vbase = i_v * VT_GDN_BV;
+
+  vt_gdn_setup(vbase);
+
+  uint srow = i_n;
+  if (p.has_idx != 0u) {
+    int si = int(IDX.v[(p.meta_off >> 2) + i_n]);
+    if (si < 0 || uint(si) >= p.n_state_rows) {  // fla NULL block; see the header
+      if (vt_gdn_vrow < p.dv && vt_gdn_wk == 0u) {
+        VT_STORE(O32, O16, VT_PC_OUT_DT, p.out_off,
+                 (i_n * p.hv + h_v) * p.dv + vt_gdn_vrow, 0.0);
+      }
+      return;
+    }
+    srow = uint(si);
+  }
+
+  vt_gdn_load_tile(srow, h_v, vbase);
+  vt_gdn_step(i_n, h_v, h_k);
+  vt_gdn_store_tile(srow, h_v, vbase);
+}

src/vt/vulkan/shaders/vt_gdn_prefill.comp

@@ -0,0 +1,78 @@
+#version 450
+// vt::GdnPrefill (kGdnPrefill) — the gated-delta-rule recurrence over a varlen
+// prompt. Ported 1:1 from src/vt/cpu/cpu_ops.cpp:1331-1366 GdnPrefillKernel; the
+// per-token arithmetic and the shared-memory state tiling both live in
+// vt_gdn_recurrence.glsl, which cites its sources.
+//
+// THE GRID IS THE CPU KERNEL'S PARALLEL AXIS, ONE STEP FINER. cpu_ops.cpp:1342-1349
+// row-chunks over the (SEQUENCE, VALUE-HEAD) product, and records why: sequences
+// alone left the whole recurrence single-threaded. This shader adds the VALUE-ROW
+// tile that our CUDA kernel already uses as its grid.x (cuda_gdn.cu:2421), so the
+// axis is (sequence, value-head, BV-value-tile) — flattened into
+// gl_WorkGroupID.x as ((s * Hv + hv) * NV + i_v) because Vulkan's guaranteed
+// workgroup-count limit is per dimension and one dimension is enough.
+//
+// THE SEQUENCE STAYS SEQUENTIAL, INSIDE THE WORKGROUP, and that is the point of
+// the shape: the [BV,Dk] state tile is staged into shared memory ONCE, run
+// through every token of this sequence, and written back ONCE. The CPU reference
+// and our CUDA decode kernel both stream the state per token; a 512-token prompt
+// touches it twice here instead of 2048 times.
+//
+// `s` is derived from gl_WorkGroupID, so the token range [qsl[s], qsl[s+1]) is
+// UNIFORM across the workgroup — required, because every step below contains
+// barriers and a barrier reached by only part of a workgroup is undefined.
+#extension GL_GOOGLE_include_directive : require
+#include "vt_common.glsl"
+
+layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in;
+
+layout(binding = 0) readonly buffer Qb32 { uint v[]; } Q32;   // q [T,Hk,Dk]
+layout(binding = 1) readonly buffer Qb16 { uint16_t v[]; } Q16;
+layout(binding = 2) readonly buffer Kb32 { uint v[]; } K32;   // k [T,Hk,Dk]
+layout(binding = 3) readonly buffer Kb16 { uint16_t v[]; } K16;
+layout(binding = 4) readonly buffer Vb32 { uint v[]; } V32;   // v [T,Hv,Dv]
+layout(binding = 5) readonly buffer Vb16 { uint16_t v[]; } V16;
+layout(binding = 6) buffer Ob32 { uint v[]; } O32;            // out [T,Hv,Dv]
+layout(binding = 7) buffer Ob16 { uint16_t v[]; } O16;
+// FOUR OPERANDS ARE BOUND THROUGH THE 32-BIT VIEW ONLY. g, beta and the state are
+// f32 BY CONTRACT on this device (src/vt/ops.cpp:1629-1643: g/beta always, and
+// the fp16/bf16 state is a CUDA-only extension), and query_start_loc is i32, so a
+// dtype-erased pair would be a branch that can never take its other side.
+layout(binding = 8) readonly buffer Gb { uint v[]; } G;       // g [T,Hv], f32
+layout(binding = 9) readonly buffer Eb { uint v[]; } BETA;    // beta [T,Hv], f32
+layout(binding = 10) buffer Sb { uint v[]; } S;               // state [N,Hv,Dv,Dk], f32
+layout(binding = 11) readonly buffer Mb { uint v[]; } QSL;    // query_start_loc [N+1], i32
+
+layout(constant_id = 0) const uint VT_PC_QKV_DT = VT_DT_F32;
+layout(constant_id = 1) const uint VT_PC_OUT_DT = VT_DT_F32;
+
+layout(push_constant) uniform Params {
+  uint hk, dk, hv, dv;
+  uint nv;            // value tiles per (sequence, head): ceil(Dv / VT_GDN_BV)
+  uint ratio;         // Hv / Hk, the GQA broadcast
+  uint has_idx;       // decode only; 0 here
+  uint n_state_rows;  // decode only; state.shape[0]
+  uint q_off, k_off, v_off, out_off, g_off, beta_off, state_off, meta_off;
+  float scale;
+} p;
+
+#include "vt_gdn_recurrence.glsl"
+
+void main() {
+  uint wg = gl_WorkGroupID.x;
+  uint i_v = wg % p.nv;
+  uint sh = wg / p.nv;
+  uint h_v = sh % p.hv;
+  uint s = sh / p.hv;
+  uint h_k = h_v / p.ratio;
+  uint vbase = i_v * VT_GDN_BV;
+
+  vt_gdn_setup(vbase);
+  vt_gdn_load_tile(s, h_v, vbase);
+  // An EMPTY sequence (qsl[s+1] == qsl[s]) runs no steps and stores the tile back
+  // unchanged, which is what the CPU reference's `for (t = qslp[s]; ...)` does.
+  int t0 = int(QSL.v[(p.meta_off >> 2) + s]);
+  int t1 = int(QSL.v[(p.meta_off >> 2) + s + 1u]);
+  for (int t = t0; t < t1; ++t) { vt_gdn_step(uint(t), h_v, h_k); }
+  vt_gdn_store_tile(s, h_v, vbase);
+}