docs(qwen35): draft TP2 phased design

docs/index.md

@@ -45,7 +45,7 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 
 | Path | TL;DR |
 | --- | --- |
-| `models/qwen35/roadmap.md` | Qwen3.5 dense roadmap v2 (#654): core correctness/admission/chunked-prefill/sampling/step-tail gates are landed; current 4B HTTP boundary is the retained #469 RTX 5090 sweep, which completed with zero failed requests but trails vLLM at high concurrency. Next: HTTP gap attribution, mixed-load ITL (#470), lifecycle recovery (#471), joint-state prefix reuse (#257), and design-first TP (#446). |
+| `models/qwen35/roadmap.md` | Qwen3.5 dense roadmap v2 (#654): Phase 1 eager dense TP2 is complete; Phase 2 is split into P2a eager unified execution and P2b local-head linear-attention/GDR sharding. CUDA Graph, recurrent snapshots, and vocabulary parallelism stay separate. |
 | `models/qwen35/load-snapshot.md` | Issue #605 publishes Qwen3.5 logical running, waiting, and KV load through the shared single-GPU/TP scheduler backend and `EngineHandle::with_load_watch`. |
 | `models/qwen35/prefix-cache.md` | Qwen3.5 prefix-cache design: a hit is valid only when full-attention KV and a complete recurrent/conv snapshot exist at the same 256-token boundary; the first version uses a fixed-budget GPU snapshot pool with joint lookup, pinning, and LRU eviction. |
 | `models/qwen35/kv-admission.md` | Issue #254 complete: Qwen3.5 now uses full-lifetime KV admission, deferred pressure handling, impossible-request rejection, explicit error semantics, direct rejection-event coverage, RTX 5090 e2e, and real HTTP pressure/post-pressure validation. |
@@ -54,7 +54,7 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | `models/qwen35/model-crate.md` | `openinfer-qwen35-4b` owns Qwen3.5 model/scheduler/recurrent ops/tests/benches; feature-gated behind `qwen35-4b` (Triton AOT is the only Python build dependency); root loads it through `EngineHandle`. Build/check/clippy, root bench sanity check, historical Qwen3.5 e2e, and scheduler e2e records live here. |
 | `models/qwen35/kernel-plan.md` | Qwen3.5-4B has a `openinfer_qwen35_4b::kernel_plan()` static descriptor mirroring the qwen3 module — enumerates every prefill/decode/unified op with its Rust call site, backend, and notes, so you can dump the active kernel mix without reading call sites. Pure refactor (issue #256), no kernel behavior change. |
 | `models/qwen35/batched-step-tail.md` | Qwen3.5 issue #353 implementation record: final prefill tail is batched, decode/unified sample from batched logits, host full-vocab copies are logprobs-only, HF + scheduler e2e pass, and final serving A/B supports only the first-token/short-output TTFT claim. |
-| `models/qwen35/tp-design.md` | Qwen3.5 TP design: Phase 1 is eager dense TP on Qwen3's controller/worker runtime; validate TP2 first, fail closed for indivisible degrees and TP+CUDA Graph, shard dense full-attention/MLP, and leave sharded linear/GDR state to follow-up. |
+| `models/qwen35/tp-design.md` | Qwen3.5 TP design: P2a adds eager `RunUnifiedStep` with `RequestId`-keyed lifecycle/artifacts and replicated GDR; P2b shards the head-indexed linear-attention/GDR surface with one hidden all-reduce after local `out_proj`. |
 | `models/qwen35/tp-implementation.md` | Qwen3.5 TP Phase 1 implementation record: eager dense TP2 worker/scheduler path, short/long HF logits gates, scheduler e2e, and real OpenAI-compatible HTTP smoke pass; remaining TP work is kept as follow-up, not a Phase 1 claim. |
 | `models/qwen35/mixed-load-itl-470.md` | Issue #470: full cold `--max-batch 8/bg=4` matrix on RTX 4090 (24/24 valid) + starvation negative control. Qwen3.5 is not immune; chunking bounds max/per-step stall but raises p99 at low QPS (~14→~80–92ms) and pulls p99/max back from the prefill wall to the chunk wall at high load; `qps·prefill_s≳1` is a throughput wall (chunking can't fix it, and ON's +15% TTFT can trip it earlier). The old "p99 immunity" was a slot-starvation artifact. |
 | `models/qwen35/adaptive-scheduler-policy.md` | Issue #727 adaptive scheduler policy record: default `off`, opt-in `auto`, hard `--max-prefill-tokens` cap, TP `auto` rejection, and pre-review whole-prefill benchmark tradeoff retained as non-default evidence. |

docs/models/qwen35/roadmap.md

@@ -47,7 +47,7 @@ out:
 | Fault isolation | Open risk: batch-level execution errors can still fail multiple active requests | #654 |
 | Prefix reuse | Open: bounded joint KV/recurrent/conv snapshot design and implementation | #257 |
 | DFlash | In flight and opt-in: correctness-first work must stay default-off until gates pass | #434, PR #626, #654 |
-| Tensor parallel | Phase 1 complete: eager dense TP2 worker/scheduler execution; Phase 2 still needs mixed-step execution and sharded linear-attention/GDR state. | `docs/models/qwen35/tp-implementation.md`, #446 |
+| Tensor parallel | Phase 1 complete: eager dense TP2 worker/scheduler execution. P2a adds eager `RequestId`-keyed unified execution with replicated GDR; P2b adds local-head linear-attention/GDR sharding. | `docs/models/qwen35/tp-implementation.md`, `docs/models/qwen35/tp-design.md`, #446 |
 
 ## Active Contract
 
@@ -133,6 +133,8 @@ Performance:
 
 ### Later
 
+- **TP Phase 2** — land P2a and P2b as separate implementation series; report a matched Phase 1 TP2 versus P2b TP2 A/B before making any performance claim.
+
 - TP for a concrete 9B or 27B TP=2 target, then 4B TP only if same-host TP=1 vs
   TP=2 shows value.
 - LoRA with real-adapter logprob parity.

docs/models/qwen35/tp-design.md

@@ -1,8 +1,8 @@
 # Qwen3.5 Tensor Parallelism Design
 
-> **TL;DR:** Qwen3.5 tensor parallelism should reuse Qwen3's controller/worker TP runtime and stay degree-parametric. Phase 1 is correctness-first eager dense TP: validate `TP=2` first, fail closed on indivisible degrees and `TP > 1` CUDA Graph, shard dense full-attention/MLP, and keep linear-attention/GDR state replicated per rank before tackling sharded GDR state.
+> **TL;DR:** Qwen3.5 TP Phase 2 is two separately delivered correctness milestones: P2a adds eager `RunUnifiedStep` with a shared ordered `RequestId` plan while retaining Phase 1 replicated GDR; P2b shards the head-indexed linear-attention/GDR surface and adds only the hidden all-reduce after local `out_proj`.
 >
-> **Last touched:** 2026-06
+> **Last touched:** 2026-07
 
 ## Goal
 
@@ -18,7 +18,7 @@ Reuse the Qwen3 TP shape:
 - `RequestId` request identity
 - coarse-grained prefill/decode/unified/drop step protocol
 - rank-local worker-owned model state
-- rank-local CUDA context, cuBLAS, graph, and NCCL resources
+- rank-local CUDA context, cuBLAS, and NCCL resources
 - hidden all-reduce after row-parallel projections
 - replicated embedding/lm_head as the first-pass simplification
 
@@ -44,28 +44,30 @@ These decisions are settled before implementation starts.
 - Each rank worker owns and mutates its own full linear-attention conv state and GDR recurrent state copy.
 - The scheduler owns logical request lifecycle and logical KV/page lifecycle only.
 - Full-attention KV is physically rank-local and sharded by local KV heads, but one logical request/page assignment is mirrored across all ranks.
-- `DropRequest`, finish cleanup, cancellation cleanup, and slot reuse must release or reset the corresponding rank-local KV/recurrent/conv state on every rank.
+- `DropRequest`, finish cleanup, cancellation cleanup, and client disconnect must release or reset the corresponding rank-local KV/recurrent/conv state on every rank by `RequestId`.
 - Qwen3.5 gated `q_proj` slicing is an explicit acceptance gate: every rank must receive both q rows and gate rows for its local query heads.
 - MLP gate/up row sharding and down column sharding require explicit reconstruction or layout tests.
 
-## Still Open / Future Discussion
+## Phase 2 Delivery Boundaries
 
-These topics should not block Phase 1 eager dense TP, but they remain design work before any later implementation.
+P2a and P2b are separate implementation series. P2a must complete its protocol and lifecycle gates before P2b changes loader, kernel, or state shapes. This keeps worker-protocol failures distinguishable from local-head loader/kernel/state failures.
 
-- TP CUDA Graph support: graph state ownership per rank, synchronized capture/replay order, NCCL capture behavior, graph padding slots, and recurrent/conv D2D slot compaction under capture.
-- Sharded linear-attention/GDR execution: local GDR AOT kernel shapes, local recurrent-state layout, local conv state layout, and Phase 2 weight slicing.
-- TP-aware prefix cache or recurrent-state snapshots.
-- Vocab-parallel embedding or `lm_head`.
+The following are separate follow-up RFCs, not Phase 2 deliverables:
+
+- TP CUDA Graph capture/replay, including graph slots, padding, synchronized capture, and recurrent/conv D2D compaction.
+- TP-aware prefix caching and recurrent-state snapshots.
+- Vocabulary-parallel embedding or `lm_head`.
 - Multi-node TP, data parallelism, and pipeline parallelism.
-- Performance optimization claims. Phase 1 is a correctness/runtime milestone, not a throughput milestone.
+
+Phase 2 is a correctness and per-rank HBM-reduction milestone. It makes no speedup promise: P2b adds one hidden all-reduce in each of the 24 linear-attention layers. Report a matched Phase 1 TP2 versus P2b TP2 A/B before making any performance claim.
 
 ## Why Dense First, GDR Second
 
 Qwen3.5 has two separable TP problems.
 
 The dense part is already proven by Qwen3: full-attention head sharding, local KV heads, MLP intermediate sharding, all-reduce after row-parallel projections, and worker-thread CUDA/NCCL execution.
 
-The linear-attention part is Qwen3.5-specific: conv state and GDR recurrent state are long-lived request state, current GDR AOT kernels are built for the global value-head shape, and slot compaction / graph padding / `DropRequest` must all preserve rank-local recurrent state. If dense TP and GDR TP land together, failures are hard to attribute. Phase 1 narrows correctness debugging to runtime + dense sharding; Phase 2 then isolates the GDR/recurrent contract.
+The linear-attention part is Qwen3.5-specific: conv state and GDR recurrent state are long-lived request state, current GDR AOT kernels are built for the global value-head shape, and `DropRequest` cleanup plus re-admission must preserve rank-local recurrent-state boundaries. If dense TP and GDR TP land together, failures are hard to attribute. Phase 1 narrows correctness debugging to runtime + dense sharding; Phase 2 then isolates the GDR/recurrent contract. CUDA Graph slot, padding, and compaction semantics remain a separate follow-up RFC.
 
 ## Architecture Summary
 
@@ -166,7 +168,7 @@ State ownership:
 - rank workers own rank-local model shards, rank-local physical KV buffers, rank-local decode buffers, and rank-local recurrent/conv state
 - rank 0 is not special for state mutation; it follows the same worker command protocol as other ranks
 - non-primary workers may return acknowledgement or step failure only, while the primary worker returns artifacts for scheduler-side result resolution
-- all workers must observe the same ordered `RunPrefillStep`, `RunDecodeStep`, `RunUnifiedStep`, `DropRequest`, and `Shutdown` commands
+- all workers must observe the same ordered `RunPrefillChunks`, `RunDecodeStep`, `DropRequest`, and `Shutdown` commands
 
 CUDA Graph:
 
@@ -180,35 +182,49 @@ Validation scope:
 - Qwen3.5 HF logits gate
 - Qwen3.5 scheduler e2e
 - long prompt / chunked prefill path
-- slot-compaction replay
-- finish/drop followed by slot reuse without stale recurrent or conv state
+- finish, explicit drop, cancellation, and client-disconnect cleanup by `RequestId`
+- subsequent admission with a new `RequestId` observes no stale KV, recurrent, or conv state
 - gated `q_proj` head-local q/gate slicing test
 - MLP gate/up shard and down shard reconstruction/layout test
 - basic TP2 serving smoke
 - startup fails closed for unsupported or indivisible degrees
 - startup fails closed for `tp_size > 1` with CUDA Graph enabled
 
-## Phase 2: Sharded Linear Attention / GDR
+## P2a: Eager TP Unified Execution
 
-Phase 2 converts linear attention from replicated execution to true TP execution.
+P2a implements eager TP `RunUnifiedStep` while retaining the Phase 1 replicated linear-attention/GDR weights, kernels, conv state, recurrent state, and scratch shapes.
 
-Shard:
+Every rank receives the same canonical `UnifiedPlan`: ordered prefill and decode items, each carrying a `RequestId` and the request-local execution inputs needed for that row. The order is the collective-order contract; every rank executes the actual plan rows in that order. P2a does not introduce CUDA Graph padded-slot semantics, D2D state movement, or a cross-rank slot-compaction protocol.
 
-- `in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`
-- `dt_bias`, `A_log`
-- conv state
-- GDR recurrent state
-- linear-attention `out_proj`
+State and artifacts are keyed by `RequestId`:
 
-Execution:
+- worker-local KV, conv, and recurrent state is found, created, promoted, and released by `RequestId`;
+- the primary worker returns prefill and decode artifacts carrying their `RequestId`;
+- the scheduler resolves artifacts by ID and rejects unknown, duplicate, or missing results instead of relying on returned row position;
+- finish, explicit drop, cancellation, and client disconnect broadcast the same `DropRequest(RequestId)` lifecycle command to every rank.
+
+The worker's internal request table may remove and reinsert entries as an implementation detail, but that is not CUDA Graph slot semantics and must not require state copying between request identities.
+
+The fixed 16-device Triton AOT handle table remains. Before model loading or worker launch, TP startup validates every requested logical CUDA ordinal against that supported range; dynamic handle allocation is not a P2a prerequisite. `tp_size > 1` with CUDA Graph requested continues to fail closed.
+
+P2a acceptance requires existing Phase 1 TP1/TP2 gates plus a TP2 mixed prefill/decode case with chunked prefill, decode completion/drop, cancellation, artifact-ID validation, lifecycle cleanup, and subsequent request admission. All work remains eager and uses the replicated GDR path.
+
+## P2b: Local-Head Linear Attention / GDR
+
+P2b converts the 24 linear-attention layers from replicated execution to true TP execution. It additionally requires `linear_num_key_heads % tp == 0` and `linear_num_value_heads % tp == 0`; unsupported degrees and unsupported local kernel shapes fail before model loading.
+
+Shard every head-indexed linear-attention/GDR surface by the local key/value-head ranges:
+
+- `in_proj_qkv`, preserving local q/k/value channel layout;
+- `in_proj_z`, `in_proj_b`, `in_proj_a`, `conv1d_weight`, `dt_bias`, and `A_log`;
+- `out_proj` input columns;
+- conv state, recurrent state, GDR scratch, and intermediate buffers.
+
+The head-dimension `norm_weight` remains deliberately replicated because it is shared by every local value head; it is not a head-indexed state or collective surface. Embedding and tied `lm_head` also remain replicated.
 
-- each rank computes local q/k/v/z/b/a
-- each rank updates only local conv state and local GDR recurrent state
-- each rank runs local gated RMSNorm/output-gate work
-- each rank runs local `out_proj`
-- all-reduce happens after `out_proj`
+Each rank runs local projections, convolution, GDR prefill/decode kernels, gated RMSNorm/output-gate work, and local `out_proj` against local dimensions. The only linear-attention collective is the hidden all-reduce after `out_proj`. Conv state and GDR recurrent state are request-local and rank-local for their full lifetime and are never all-reduced or centralized.
 
-Never all-reduce GDR recurrent state or conv state. Their ownership is rank-local and request-local.
+P2b acceptance requires loader reconstruction/layout tests, local AOT-kernel shape validation, rank-local allocation checks, Phase 1 and P2a regression gates, short and long TP2 HF replay, cleanup without stale request state, and a matched Phase 1 TP2 versus P2b TP2 HBM/latency/throughput report. The report is evidence, not a speedup threshold.
 
 ### vLLM Reference
 
@@ -221,15 +237,15 @@ Use vLLM's `Qwen3NextForCausalLM` / `QwenGatedDeltaNetAttention` as the referenc
 - b/a projections are local-value-head aware; some quantized paths may replicate small projections and slice locally
 - GDR prefill/decode kernels consume local head/state shapes
 
-OpenInfer-specific work remains: worker-owned rank-local recurrent state, `RequestId` lifecycle, local-state slot compaction, `DropRequest` cleanup, and fail-closed kernel-shape validation.
+OpenInfer-specific work remains: worker-owned rank-local recurrent state, `RequestId` lifecycle, request-state removal and re-admission, `DropRequest` cleanup, and fail-closed kernel-shape validation.
 
 Validation scope:
 
 - Phase 1 gates still pass
 - long HF logits replay under the validated degree
-- slot compaction replay
-- recurrent-state cleanup on finish/drop
-- no stale local recurrent state after slot reuse
+- request-state cleanup and re-admission replay
+- recurrent-state cleanup on finish/drop/cancellation
+- no stale local recurrent state after a new `RequestId` is admitted
 
 ## References
 

docs/models/qwen35/tp-implementation.md

@@ -136,31 +136,33 @@ Stable test knobs:
 - `OPENINFER_TEST_TP_DEVICES`: comma-separated TP2 CUDA ordinals. Defaults to `0,1`; examples: `1,2`, `2,3`. TP2 tests require exactly two distinct ordinals.
 - `OPENINFER_TEST_FRONTEND_MODEL_PATH`: optional tokenizer/config metadata path for HTTP serving tests. Defaults to `OPENINFER_TEST_MODEL_PATH` when unset.
 
-## Follow-Up Work
+## Phase 2 Follow-Up
 
-The exact Phase 2 split is not decided yet. The items below are retained as follow-up work that should be scoped in the design branch before implementation.
+Phase 2 is locked in `docs/models/qwen35/tp-design.md` as two separate implementation series: P2a is eager mixed unified execution on the replicated Phase 1 GDR path; P2b shards the head-indexed linear-attention/GDR weight and state surface. P2a protocol/lifecycle gates must complete before P2b changes loader, kernel, or state shapes.
 
-### TP mixed-step unified execution
+### P2a: TP mixed-step unified execution
 
-Implement `RunUnifiedStep` under TP while keeping Phase 1's replicated linear-attention/GDR state unless the design branch decides otherwise.
+Implement eager `RunUnifiedStep` under TP while retaining Phase 1's replicated linear-attention/GDR weights, kernels, conv state, recurrent state, and scratch shapes.
 
 Goals:
 
 - Support mixed prefill+decode scheduler steps under TP.
 - Preserve deterministic collective ordering across ranks.
 - Return mixed prefill/decode artifacts from the primary rank.
+- Use the `RequestId`-keyed `UnifiedPlan` and artifact contract in `tp-design.md`; P2a does not introduce CUDA Graph padded-slot or compaction semantics.
 - Validate finish/drop/client-disconnect cleanup under mixed-step execution.
 - Keep TP CUDA Graph disabled unless a separate graph design is completed.
+- Keep the fixed 16-device Triton AOT handle table and add startup-time validation for unsupported logical CUDA ordinals.
 
 Why this should be separated from GDR sharding:
 
 - Mixed-step scheduling is an execution-protocol problem.
 - Sharded linear-attention/GDR is a model-state-shape problem.
 - Combining them would make failures hard to attribute.
 
-### Sharded linear-attention/GDR state
+### P2b: sharded linear-attention/GDR state
 
-Shard the Qwen3.5 linear-attention/GDR path after the mixed-step and state-lifecycle contract is clear.
+Shard the Qwen3.5 linear-attention/GDR path after P2a establishes the mixed-step and state-lifecycle contract.
 
 Expected work:
 
@@ -169,6 +171,7 @@ Expected work:
 - adapt or regenerate GDR kernels for local state shapes
 - keep recurrent/conv state rank-local and request-local
 - all-reduce only after local linear-attention `out_proj`
+- report matched Phase 1 TP2 versus P2b TP2 HBM/latency/throughput data before making a performance claim
 
 Non-negotiable invariant: