[Model] Qwen3.5 dense roadmap v2: Stable single-GPU serving

[CAICAIIs]

Crate: openinfer-qwen35-4b · Roadmap doc: docs/models/qwen35/roadmap.md · Project roadmap: #492 · Supersedes: #249

Qwen3.5 dense serving has completed the bring-up stage: 4B, 9B, and 27B have
short/long HF logits gates; full-lifetime KV admission, direct paged prefill,
scheduler-level chunked prefill, batched sampling, and batched step-tail
execution are landed.

The next stage is promotion from Maturing toward Stable. Availability
and performance move in parallel: retain a real HTTP lifecycle gate, make the
mixed-load benchmark valid, close the measured 4B concurrency gap, and keep
hybrid KV/recurrent state correct across prefix reuse and speculation. This
roadmap stays inside the dense BF16 single-GPU product boundary.

Current state

Area	State
Accuracy	✅ 4B / 9B / 27B short and long HF bf16 logits gates
Admission and long context	✅ full-lifetime KV accounting and explicit context/KV rejection
Prefill	✅ direct paged writes, bounded scheduler chunking, and resumed base_pos > 0 coverage
Sampling and step tail	✅ mixed batched sampling and batched final norm/lm_head/token selection
4B serving performance	❌ retained RTX 5090 1024/256 output throughput trails vLLM 0.23.0 by about 10% at c1 and 39% at c16
Serving overhead	❌ c16 direct TPOT is about 9.32ms; HTTP TPOT is about 15.57ms
Mixed-load evidence	⚠️ the harness can starve injected prefill when max_batch == background_concurrency (#470)
HTTP lifecycle	❌ cancel, disconnect, overload, rejection, and recovery are not retained as one gate (#471)
Fault isolation	⚠️ several batch-level execution errors can fail all active requests
Prefix cache	❌ joint KV plus recurrent/conv snapshot implementation remains open (#257)
DFlash	⚠️ #434 / PR #626 are the active opt-in correctness-first DFlash work
Tensor parallel	❌ the engine currently accepts one CUDA device; design issue #446 is open

Now

• qwen35: retain current serving benchmark sweep vs vLLM #469 — retain the current 4B OpenInfer-vs-vLLM serving matrix with
  prefill-heavy, decode-heavy, concurrency, QPS, overload, tail-latency,
  failure-count, and output-sanity evidence.
• qwen35: measure mixed-load ITL after scheduler-level chunked prefill #470 — make the mixed-load ITL gate prove that injected prefill overlaps
  active decode; keep the starvation setup as a negative control.
• qwen35: add HTTP reliability gate under cancel, disconnect, and overload #471 — retain HTTP cancel, disconnect during prefill/decode, explicit
  rejection, admissible overload, post-pressure completion, health, and GPU
  memory-return evidence.
• Finish the PR feat(qwen35): add opt-in DFlash speculative decoding #626 DFlash correctness gate in parallel: accept-all,
  accepted-prefix, reject-first, fallback, long-context, memory reservation,
  and joint KV/recurrent/conv restore.

Next

• Attribute the direct-vs-HTTP gap across queue wait, scheduler planning,
  request/event delivery, sampling/logprob copies, GPU step time, and idle gaps.
• Close the primary 4B c1/c4/c8/c16 and QPS gap with repeated same-host serving
  runs. Promote only changes that improve E2E serving without regressing
  correctness, lifecycle recovery, or mixed-load ITL.
• Revisit unified versus split prefill/decode execution only after the retained
  mixed-load traces identify the limiting path.
• Profile the remaining kernel frontier before changing it: GDN projections,
  QK-norm/RoPE/KV-write fusion, uncompiled GQA groups, and per-shape cublasLt
  tuning.
• qwen35: prefix caching needs a design for recurrent state (discussion) #257 — implement one bounded prefix-cache facade that commits and restores
  full-attention KV plus recurrent/conv snapshots at the same token boundary.
• After DFlash correctness merges, open a fresh performance child issue only if
  a retained profile shows a concrete single-stream, graph, or c1 bottleneck.
  DFlash does not block default-serving maturity.

Later

• qwen35: design tensor parallelism for hybrid recurrent state #446 — choose a concrete 9B or 27B TP=2 checkpoint and hardware target, then
  settle KV, recurrent state, conv state, projection, MLP, embedding/lm_head,
  and communication ownership.
• Add 4B TP only when a same-host TP=1-vs-TP=2 run shows measurable value.
• LoRA with real-adapter logprob parity.
• Exporter-specific FP8/NVFP4/MXFP4 loading and accuracy lanes.
• MTP/DSpark as a separate speculative path.
• Hybrid-state offload and P/D under a separate transfer/restore contract.
• MoE and VLM under separate model roadmaps.

Active contract

• The supported product boundary in this roadmap is dense BF16 Qwen3.5
  4B/9B/27B on one GPU. 4B is the serving-performance anchor.
• 9B and 27B correctness do not inherit 4B performance or production-readiness
  claims.
• Qwen3.5 state is transactional: paged full-attention KV, recurrent f32 state,
  conv bf16 state, and logical position move, cache, roll back, or transfer
  together.
• The primary performance contract is repeated same-host OpenInfer-vs-vLLM
  HTTP serving with pinned versions, identical clients, raw artifacts, output
  sanity, completed/failed counts, TTFT, TPOT, ITL p99, and output throughput.
• Direct diagnostics, profiler attribution, microbenchmarks, HTTP pressure, and
  serving claims stay labeled separately.
• A performance improvement does not establish production readiness.
  Availability requires lifecycle recovery, bounded memory, explicit failures,
  post-pressure health, and retained evidence.
• DFlash remains opt-in and default-off until its correctness and tail-behavior
  gates pass.

Done criteria

• Cancel, disconnect, overload, rejection, and a clean follow-up request remain
  green without wedges, silent request loss, or leaked request state.
• Every retained mixed-load cell proves the intended prefill/decode overlap and
  records saturated, rejected, failed, or OOM outcomes instead of dropping
  them.
• 4B c1/c4/c8/c16 and QPS output throughput and TPOT match or beat the pinned
  vLLM baseline inside the documented RTX 5090 workload envelope.
• Short/long HF gates remain green for 4B/9B/27B across every enabled execution
  mode.
• Prefix reuse restores one valid joint KV/recurrent boundary, survives
  eviction and memory pressure, and preserves cold-serving behavior when the
  snapshot pool cannot accept an insertion.
• DFlash, when enabled, passes real GPU correctness and same-commit A/B gates;
  it may remain deferred without blocking default-serving maturity.
• Unsupported model size, context, sampling, topology, cache, and speculative
  combinations fail explicitly.
• No open blocker remains for the documented single-GPU dense Stable claim.

Existing open issues #257, #434, #446, #469, #470, and #471 remain the source
of truth for their implementation contracts. Open new child issues only when a
retained failure cell or profile identifies one concrete owner and acceptance
test. Closed historical issues should stay out of the sub-issue list.

Close #249 with a pointer here after this roadmap is accepted.

[github-actions]

This issue has been inactive for 14 days. It will be closed after another 30 days unless there is new activity.