feat(qwen35): add opt-in DFlash speculative decoding

docs/index.md

@@ -46,6 +46,7 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | Path | TL;DR |
 | --- | --- |
 | `models/qwen35/roadmap.md` | Qwen3.5-4B roadmap (2026-06 review): decode-tuning refresh improves direct TPOT by 2-3%, while vLLM still leads 1024/256 HTTP decode and high-concurrency throughput. Open items: HND prefill staging, prefix-cache design, serving concurrency. |
+| `models/qwen35/dflash-speculative-decoding.md` | Qwen3.5 DFlash speculative decoding is opt-in behind `--dflash-draft-model-path`; the current implementation supports a single-active greedy path with normal-decode fallback for multi-active/logprobs and no concurrent-throughput claim. |
 | `models/qwen35/kv-admission.md` | Issue #254 complete: Qwen3.5 now uses full-lifetime KV admission, deferred pressure handling, impossible-request rejection, explicit error semantics, direct rejection-event coverage, RTX 5090 e2e, and real HTTP pressure/post-pressure validation. |
 | `models/qwen35/optimization.md` | Hybrid 24 linear + 8 full attn optimization ledger. Decode-tuning refresh fuses MLP gate/up and tunes decode cublasLt buckets, improving direct TPOT by 2-3%; vLLM still leads 1024/256 HTTP decode. |
 | `models/qwen35/accuracy.md` | Qwen3.5 HF bf16 logits goldens, size-keyed (4b, 9b committed; 27b once dumped), through `past_key_values`: short replay covers sequential graph, bucket-straddling batched graph, and slot-compaction; long replay covers 4097/8192-token prompts; full GSM8K 8-shot now matches the HF baseline within 0.15 percentage points. |

docs/models/qwen35/dflash-speculative-decoding.md

@@ -0,0 +1,90 @@
+# Qwen3.5 DFlash Speculative Decoding
+
+> **TL;DR:** Qwen3.5 DFlash is an opt-in, single-active greedy path behind `--dflash-draft-model-path`. On RTX 5090, the verified path improves direct output throughput by 2.08x at prompt 64 / output 128 and 2.40x at prompt 1024 / output 256, with matching output hashes. Multi-active and unsupported request shapes use normal decode.
+
+Last touched: 2026-07
+
+## Design
+
+Qwen3.5 can load a DFlash draft model beside the target model. The default path is unchanged when no draft path is provided.
+
+Speculative verification is a transaction over all Qwen3.5 decode state:
+
+- paged full-attention KV;
+- linear-attention recurrent state;
+- convolution state and sequence length.
+
+Verification writes recurrent and convolution state into scratch buffers. A fully accepted span keeps the verified KV and copies the verified recurrent state into the live slot. A partial acceptance truncates KV, restores the backed-up recurrent state, and replays only the accepted span. Partial replay disables timing-selected cuBLASLt algorithms because those algorithms caused rare greedy hash drift on this shape; normal decode and batched verification keep the tuned cuBLASLt path.
+
+## Enable
+
+```bash
+OPENINFER_TRITON_PYTHON=<triton-python> \
+cargo run --release --features qwen35-4b -- \
+  --model-path <Qwen3.5-4B> \
+  --dflash-draft-model-path <Qwen3.5-DFlash>
+```
+
+The speculative path requires:
+
+- one active request;
+- greedy sampling with `logprobs=0`;
+- captured DFlash prompt context;
+- a complete verify span within the 2048-token validated context;
+- at least two output tokens remaining.
+
+Multi-active, non-greedy, logprobs, missing-context, and longer-context requests continue through normal target decode. Once normal decode takes ownership, the captured draft state is discarded so a request cannot later resume speculation with stale context. LoRA, KV offload, tensor parallel, and decode overlap are rejected when DFlash is enabled.
+
+## Validation
+
+The GPU-only tests are ignored by default, so `--ignored` is required:
+
+```bash
+OPENINFER_TRITON_PYTHON=<triton-python> \
+OPENINFER_TEST_MODEL_PATH=<Qwen3.5-4B> \
+cargo test --release -p openinfer-qwen35-4b --features qwen35-4b \
+  --test speculative_verify -- --ignored --nocapture --test-threads=1
+
+OPENINFER_TRITON_PYTHON=<triton-python> \
+OPENINFER_TEST_MODEL_PATH=<Qwen3.5-4B> \
+OPENINFER_DFLASH_TEST_MODEL_PATH=<Qwen3.5-DFlash> \
+cargo test --release -p openinfer-qwen35-4b --features qwen35-4b \
+  --test dflash_speculative_gate -- --ignored --nocapture --test-threads=1
+```
+
+RTX 5090 results:
+
+- `speculative_verify`: 6 passed;
+- `dflash_speculative_gate`: 3 passed;
+- `e2e_scheduler`: 1 passed;
+- pinned short and long `hf_golden_gate`: 2 passed;
+- independent-process stability: eager 20/20 and CUDA Graph 20/20, all with output hash `5a71ac0dfe1cd1e5`.
+
+## Direct Benchmark
+
+Same GPU, model revision, source tree, CUDA Graph setting, and benchmark client. The prompt 64 / output 128 row is the median of three alternating-order A/B runs with warmup 5 and 20 measured iterations per run.
+
+Environment: RTX 5090, driver 580.76.05, CUDA 12.8, Rust 1.96.1, Triton 3.7.1, target revision `851bf6e806ef`, and `z-lab/Qwen3.5-4B-DFlash` config SHA-256 prefix `6fa9ca0d10d2`.
+
+| Shape | Path | Baseline output tok/s | DFlash flag output tok/s | Delta | Output |
+| --- | --- | ---: | ---: | ---: | --- |
+| prompt 64 / output 128 / c1 | speculative | 158.33 | 329.90 | +108.36% | hash match |
+| prompt 1024 / output 256 / c1 | speculative | 136.85 | 327.90 | +139.60% | hash match |
+| prompt 1 / output 256 / c1 | normal fallback | 159.22 | 158.57 | -0.40% | hash match |
+| prompt 4096 / output 256 / c1 | normal fallback | 101.14 | 100.69 | -0.44% | hash match |
+
+For prompt 64 / output 128, median end-to-end latency falls from `808.48 ms` to `388.03 ms`; TTFT remains flat at `11.68 ms` versus `11.28 ms`, and steady TPOT falls from `6.27 ms` to `2.86 ms`.
+
+The single-request reservation costs 1,817 MB of fixed GPU memory on this host. Target KV capacity remains 31,098 pages versus 34,188 without DFlash, retaining about 91% of the baseline page pool. The earlier pool-scaled draft reservation retained only 10,438 pages and was removed.
+
+Nsight Systems on the same shape attributes the gain to less target work:
+
+- total GPU kernel time: about `790 ms` to `323 ms`;
+- device-to-host API time: `810 ms` to `249 ms`;
+- partial-commit GemmEx replay: `6.58 ms`, about `2%` of DFlash GPU kernel time.
+
+The partial replay path is not the leading GPU or tail bottleneck. The remaining DFlash GPU time is dominated by the expected batched target-verification GEMMs.
+
+## Claim Boundary
+
+These are direct single-active results, not HTTP serving or vLLM parity results. c4/c8/c16 currently use normal decode fallback, so this implementation does not claim concurrent speculative throughput improvement.

docs/models/qwen35/roadmap.md

@@ -1,8 +1,8 @@
 # Qwen3.5-4B Roadmap
 
-> **TL;DR:** Qwen3.5-4B is decode-correct and still improving: the decode-tuning refresh improves direct TPOT by `2.1-3.2%`, while vLLM still leads 1024/256 HTTP decode and high-concurrency throughput. Long-prompt HF logits and GSM8K gates cover the old 4096-position RoPE boundary. Remaining structural items are HND prefill staging, prefix-cache design, and the serving-level concurrency gap.
+> **TL;DR:** Qwen3.5-4B is decode-correct and still improving: the decode-tuning refresh improves direct TPOT by `2.1-3.2%`, while vLLM still leads 1024/256 HTTP decode and high-concurrency throughput. Long-prompt HF logits and GSM8K gates cover the old 4096-position RoPE boundary. Opt-in single-active DFlash is correctness-gated and shows 2.08x-2.40x direct throughput on verified shapes; multi-active speculation remains follow-up work. Remaining structural items are HND prefill staging, prefix-cache design, and the serving-level concurrency gap.
 >
-> **Last touched:** 2026-06
+> **Last touched:** 2026-07
 
 Tracking issue: see the `[Model] Qwen3.5-4B roadmap` GitHub issue. Sibling doc: `docs/models/qwen3/roadmap.md` — batched sampling is shared and #284 now routes Qwen3.5 decode through the same compact batched sampler; Qwen3.5 now has its own model-level non-greedy behavior gate, while qwen3 keeps the sibling gate on its side.
 
@@ -20,6 +20,7 @@ Tracking issue: see the `[Model] Qwen3.5-4B roadmap` GitHub issue. Sibling doc:
 | Admission | ✓ existing full-lifetime KV admission and explicit `Rejected` events cover impossible KV requests; #253 adds the context-window rejection reason before prefill/decode | `scheduler.rs`, `src/scheduler/plan.rs`, `docs/models/qwen35/kv-admission.md` |
 | Scheduler tests | Partial: current plan selection, full-lifetime admission, context-window rejection, slot assignment, and slot-compaction decisions are CPU-tested; GPU execution remains coupled to the production scheduler | `src/scheduler/plan.rs` |
 | Step tail | Local branch verified: #353 batches the prefill final norm/lm_head tail, samples decode/unified rows from batched logits, and keeps host full-vocab copies only for requested logprobs; HF/e2e gates pass, short-output serving A/B shows TTFT benefit, long-decode TPOT remains a no-claim diagnostic | `docs/models/qwen35/batched-step-tail.md` |
+| DFlash | Opt-in single-active greedy DFlash path behind `--dflash-draft-model-path`; verified direct c1 shapes improve 2.08x-2.40x with matching hashes, while multi-active/logprobs fall back to normal decode | `docs/models/qwen35/dflash-speculative-decoding.md` |
 | TP | ✗ absent (single GPU only) | — |
 | Prefix cache | ✗ absent; recurrent GDR state (~48MB per boundary snapshot) makes "prefix hit" itself a design question | — |
 

openinfer-core/src/kv_pool.rs

@@ -213,6 +213,20 @@ impl KvState {
         self.seq_len += count;
     }
 
+    /// Roll this request's logical KV length back to `token_count`, returning
+    /// any now-unused tail pages to the pool.
+    pub fn truncate_to(&mut self, token_count: usize) -> Result<()> {
+        anyhow::ensure!(
+            token_count <= self.seq_len,
+            "KvState cannot truncate from {} up to {token_count}",
+            self.seq_len
+        );
+        let needed = pages_needed(token_count, self.pool.inner.layout.page_size);
+        self.permit.truncate(needed);
+        self.seq_len = token_count;
+        Ok(())
+    }
+
     /// Build kernel-facing metadata for this request's KV.
     pub fn desc(&self) -> KvDesc<'_> {
         KvDesc {
@@ -348,12 +362,39 @@ mod tests {
         assert_eq!(desc.last_page_len(), 1);
         assert_eq!(pool.available_pages(), 2);
 
+        // Truncate back into the first page: tail page returns immediately.
+        kv.truncate_to(15).unwrap();
+        assert_eq!(kv.seq_len(), 15);
+        let desc = kv.desc();
+        assert_eq!(desc.num_pages(), 1);
+        assert_eq!(desc.last_page_len(), 15);
+        assert_eq!(pool.available_pages(), 3);
+
+        // Truncate to zero releases all request pages.
+        kv.truncate_to(0).unwrap();
+        assert_eq!(kv.seq_len(), 0);
+        assert_eq!(kv.desc().num_pages(), 0);
+        assert_eq!(pool.available_pages(), 4);
+
         // Reset returns all pages
+        kv.ensure_capacity(17).unwrap();
+        kv.advance(17);
         kv.reset();
         assert_eq!(kv.seq_len(), 0);
         assert_eq!(pool.available_pages(), 4);
     }
 
+    #[test]
+    fn kv_state_rejects_truncate_forward() {
+        let pool = test_pool(16, 3);
+        let mut kv = pool.alloc();
+        kv.ensure_capacity(4).unwrap();
+        kv.advance(4);
+
+        let err = kv.truncate_to(5).unwrap_err().to_string();
+        assert!(err.contains("cannot truncate from 4 up to 5"));
+    }
+
     #[test]
     fn kv_state_out_of_pages() {
         // 3 pages total: 1 padding, 2 available → 32 tokens max

openinfer-core/src/ops/paged_plan.rs

@@ -11,6 +11,20 @@ pub struct PrefillPagedPlan {
 }
 
 impl PrefillPagedPlan {
+    pub fn estimate_preallocated_bytes(
+        max_total_tokens: usize,
+        max_total_pages: usize,
+        max_batch: usize,
+        max_tiles: usize,
+    ) -> usize {
+        openinfer_kernels::ops::PrefillPagedPlan::estimate_preallocated_bytes(
+            max_total_tokens,
+            max_total_pages,
+            max_batch,
+            max_tiles,
+        )
+    }
+
     pub fn new(
         ctx: &DeviceContext,
         desc: &KvDesc<'_>,

openinfer-core/src/page_pool.rs

@@ -107,6 +107,24 @@ impl OwnedPagePermit {
         }
         true
     }
+
+    /// Return tail pages until the permit holds exactly `new_len` pages.
+    ///
+    /// Prefix page order is preserved. Pages beyond `new_len` are returned to
+    /// the same pool immediately, matching the drop-time LIFO reuse order.
+    pub(crate) fn truncate(&mut self, new_len: usize) {
+        assert!(
+            new_len <= self.pages.len(),
+            "cannot grow an OwnedPagePermit via truncate"
+        );
+        if new_len == self.pages.len() {
+            return;
+        }
+
+        let returned = self.pages.split_off(new_len);
+        let mut free_list = self.inner.free_list.lock();
+        free_list.extend(returned.into_iter().rev());
+    }
 }
 
 impl Drop for OwnedPagePermit {
@@ -182,4 +200,27 @@ mod tests {
         // all 4 pages back after drop
         assert_eq!(pool.available_pages(), 4);
     }
+
+    #[test]
+    fn truncate_returns_tail_pages_and_preserves_prefix() {
+        let pool = PagePool::new(5);
+
+        {
+            let mut permit = pool.try_acquire_many(4).expect("initial acquire");
+            assert_eq!(
+                permit.pages(),
+                &[PageId(0), PageId(1), PageId(2), PageId(3)]
+            );
+            assert_eq!(pool.available_pages(), 1);
+
+            permit.truncate(2);
+            assert_eq!(permit.pages(), &[PageId(0), PageId(1)]);
+            assert_eq!(pool.available_pages(), 3);
+
+            let next = pool.try_acquire_many(2).expect("tail pages reusable");
+            assert_eq!(next.pages(), &[PageId(2), PageId(3)]);
+        }
+
+        assert_eq!(pool.available_pages(), 5);
+    }
 }