Qwen3.5-4b BF16 optimization and correctness (sm_120)

[richiejp]

Refs #206
Closes #237

Row

KERNEL-SSM-MAMBA — Qwen3.5-4B BF16 optimization and correctness work for consumer Blackwell (sm_120).

Before starting

• Issue/PR search and existing claim: RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT, and VRAM gaps vs vLLM #206 tracks the remaining RTX 5070 Ti performance gaps. CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237 was discovered while rebasing and testing this branch. No existing PR
  owns this branch.
• Roadmap or matrix row: ROAD-V1-C2-LOCAL-BF16, KERNEL-SSM-MAMBA, with supporting benchmark work under SERVE-CLI-BENCH.
• Anchors inspected: the pinned vLLM Qwen3.5 execution path, local GDN kernels, benchmark admission/frontend path, CUDA BF16 MoE resident cache, and the existing
  Qwen3.5-4B benchmark evidence.

What changed

Improves the Qwen3.5-4B BF16 path on an RTX 5070 Ti (sm_120) while preserving exact output. The branch adds pretokenized benchmark input and atomic request-wave
admission, retains the accepted GDN optimizations, records and removes rejected experiments, and updates the benchmark evidence.

It also fixes #237: the CUDA BF16 MoE resident cache was keyed by a reusable MoeBlockWeights wrapper address and could return stale device pointers after engine
teardown. The cache is now keyed by the uploaded router allocation and retains its allocation owner. A deterministic repeated-engine regression covers both identity
reuse and owner lifetime.

Evidence

• scripts/agent-preflight.sh passes

  • Full preflight still reports established unrelated baseline failures, including test-registration/live-row checks and sandboxed release-fixture failures.
  • Commit trailers, document checkpoint, NOW checkpoint, diff check, and PR-size checks pass.

• Focused CUDA gate passes at 1d8cfcbed33868bda50f6be66b2f3655cbae545e:

  nix develop .#cuda --command flock /tmp/gpu -c \
    "ctest --test-dir build-nix-cuda --output-on-failure \
    -R '^(test_engine_core_proc|test_async_llm|test_bench|test_gdn_decode_fused|test_gdn_prefill_conv|test_ops_gdn)$'"
  

Result: 6/6 passed.

• test_bench repeated-process stress: 50/50 passed.

• Independent mutation review:

  • Restoring the old MoeBlockWeights* cache key fails with invalid identities and zero-token output.
  • Removing the retained allocation owner fails with 20 invalid cache entries.
  • Restored implementation passes; reviewer verdict: PASS with no findings.

• Same-change records are updated in docs/STATUS.md, docs/BENCHMARKS.md, docs/FEATURES.md, .agents/NOW.md, and the owning specs.

Speed claims

• The operator ran the measurements under the GPU lock and recorded them in docs/BENCHMARKS.md.

Corrected Qwen3.5-4B BF16 comparison against production vLLM:

• Total/output throughput: 1.0283x vLLM — 2.83% ahead.
• TTFT: 1.0853x slower.
• TPOT: 1.0165x slower.
• E2E latency: 1.0288x slower.
• VRAM: 118.7 MiB higher.

Honest gaps

• TTFT, TPOT, E2E latency, and VRAM parity remain open under RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT, and VRAM gaps vs vLLM #206.
• The retained GDN optimizations improve the measured timing axes but do not close every gap.
• Rejected argmax-scratch, vector-writeback, and pre-serve-warmup experiments are documented and removed from product code.
• Full repository preflight remains red on pre-existing unrelated checks; no waiver was added.
• The process-lifetime BF16 MoE cache retains one small router allocation for each cached engine identity.