Qwen3.5-4b BF16 optimization and correctness (sm_120)

.agents/NOW.md

@@ -21,7 +21,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`.
 | MiniMax-H3 | **PRUNED ckpts RUN (#241): Q8_0 renders, seam 0.9941** | same-binary A/B |
 | Kimi-Linear-48B | 122/128 held; grouped router parallelised, e2e NOT ESTABLISHED | ckpt is tiktoken-only: no warm server |
 | 35B binding grid | @`a0fa12c7`: **flat 0.935-0.979x, NO c2/c8 weak cell** (CoV <0.81%); mem PSS 3.81x | Attribute the flat ~5% mid-band; TTFT c2 0.872x |
-| Qwen3.5-4B sm_120 | Exact chunks ON: 3.072x kernel / +2.272% run; sealed-vLLM tput 1.021x PASS; latency/VRAM OPEN | Spike residual 1.609x conv gap |
+| Qwen3.5-4B sm_120 | Exact; tput **1.0283x**. Slower: TTFT/TPOT/E2E **1.0853/1.0165/1.0288x**; VRAM +118.7 MiB | Profile first wave |
 | RPi5 A76 CPU | **R5 asm GREEN; llama NOT MET**: 0.461x pf, 0.653x dec | W6: BF16 GEMM |
 | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED** | TERMINAL: at parity |
 | SERVE-ASYNC-DENSE-MIRROR | **LANDED+VERIFIED** (`f9c969ae`): async mirror, dense Qwen3; SACRED 184/184 | Sibling scope one-liner |
@@ -53,8 +53,8 @@ latency/memory on every axis, both gate models, reproduced 2–3x idle. See
    of roof. Dense-marlin +0.5%; Triton-AOT GDN a WASH.
 2. **Spike the Parakeet encoder row** (vLLM: `nano_nemotron_vl.py`; the
    transducer half is NOT in vLLM: separate call).
-3. **Qwen3.5-4B sm_120:** GREEN + reprofiled. Spike the residual 1.609x conv
-   gap; latency/VRAM and gate models stay open.
+3. **Qwen3.5-4B #206:** exact, tput +2.83%; latency/VRAM open. Profile first
+   wave; argmax/writeback closed.
 2. **Merge the invocation-parity prevention** (CI guard + AGENTS.md checklist);
    CUDA build-verify the byte-exact `kGemvHeuristicAlgos` refactor on dgx.
 4. **Restore `local-ai-worker`** on dgx at campaign end (`--restart=always`).

.agents/engine-matrix.md

@@ -204,7 +204,7 @@ claims it.
 | `SERVE-HTTP-TRANSPORT` | Serving-socket transport parity: mirror vLLM's uvicorn/asyncio default `TCP_NODELAY` on every accepted SSE socket so per-token stream frames are not held by Nagle against the peer's delayed ACK. Implemented + CPU-tested; the non-binding localhost A/B sizing is COMPLETE and NEUTRAL within noise on c1/c2 ITL/TPOT/throughput (loopback ACKs are instant, so Nagle never coalesces ~100 ms-cadence token frames) — no gate-axis credit expected; the mirror stays for real-network parity. Future keep-alive / read-write-timeout / listening-socket option parity noted, not done | T0 | vLLM serves via uvicorn over asyncio `vllm/entrypoints/launcher.py:71,76`, `vllm/entrypoints/openai/api_server.py:591,630`; asyncio disables Nagle per accepted TCP stream socket `asyncio/base_events.py:192-197` (`_set_nodelay`) called from `asyncio/selector_events.py:950`; cpp-httplib default-off `third_party/httplib/httplib.h:142`, applied on accept only when set `third_party/httplib/httplib.h:12083` | `src/vllm/entrypoints/openai/api_server.cpp:69` (`set_tcp_nodelay(true)` in the ApiServer setup) | behavioral accepted-socket `getsockopt(TCP_NODELAY)` case `tests/vllm/entrypoints/openai/test_api_server.cpp:1076` (helper `:380`); RED accepted `TCP_NODELAY` 0 → GREEN 1, full `test_openai_api_server` **22/22 cases / 242 assertions**; non-binding sizing root `~/work/vllm.cpp-tcpnodelay-sizing/ff915e8…` (raw-set SHA `f5b52900…2128`) neutral within noise; closure [ledger](parity-ledger.md#L451) | [serve-tcp-nodelay.md](specs/serve-tcp-nodelay.md) | `DONE` | `ff915e8` |
 | `SERVE-C-ABI` | Stable LocalAI-style C FFI (**19** exported `VLLM_API` symbols at `VLLM_ABI_VERSION 10`; blocking and nonblocking request handles. Count corrected 2026-07-24 from a stale `17`, which predated ABI v4/v5 adding `tool_parser`/`reasoning_parser` and the chat entry points; `include/vllm.h` is the source of truth and README:231 already said 19). **ABI v9 2026-07-28 (`CLAIM-CAPI-ENGINE-CONFIG-V9`): the ABI carried strictly LESS engine config than `EngineParams` does** - `max_num_batched_tokens`, the scheduler `scheduling_policy` (`fcfs` / `priority` / `lpm`), and `kv_transfer_config` (the external KV connector / LMCache JSON) were reachable from the bundled server's flags and from NO embedder. All three added, inert at their defaults (zero-filled v8 growth == byte-identical pre-v9 engine); the connector NAME is validated against `KVConnectorFactory` at load, mirroring the server's startup check. `tokenizer_config_path` stopped being a declared-since-v1 no-op and now selects the chat template's source file. Malformed `speculative_config`/`kv_transfer_config` documents now report `VLLM_ERR_INVALID_ARGUMENT` (the contract vllm.h documented since v6) instead of `VLLM_ERR_MODEL_LOAD`, via a catch scoped to the parse block so a real `FromModelDir` failure still reports MODEL_LOAD. Driver: the LocalAI vllm-cpp backend could not expose LMCache or the prefill budget in a model config) | T0 | Original project ABI; pinned vLLM has no C ABI | `include/vllm.h:143,181,207`; `src/capi/vllm_c.cpp:229,264,327,391` | `tests/capi/test_capi.cpp:320,428,505,574,606,640`; `tests/capi/test_dlopen.cpp:77,86`; `tests/capi/c_header_compile.c:1` | [c-api-library.md](specs/c-api-library.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-C-ABI-SPIKE` |
 | `SERVE-CPP-API` | Rich `LLM` and `AsyncLLM` C++ API | T1 | `vllm/entrypoints/llm.py:66,422`; `vllm/v1/engine/async_llm.py:70` | - | - | `planned: specs/cpp-api.md` | `INVENTORIED` | - |
-| `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29`; production queue `vllm/v1/engine/core.py:200-231,622-669` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; production `AsyncLLM` benchmark frontend + auditable scheduler depth `examples/bench/bench_core.h:426,495,595`; `examples/bench/main.cpp:51` | server help contract `examples/CMakeLists.txt:34`; production-frontend and metric assertions `tests/examples/test_bench.cpp:18,29-32,61,81,97` | [CLI/serve/benchmark spike](specs/cli-serve-bench.md) | `PARTIAL` | - |
+| `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29`; production queue `vllm/v1/engine/core.py:200-231,622-669`; pinned comparison pretokenizes before timing and synchronously admits a complete concurrency wave before each explicit step `tools/bench/vllm_closed_loop_metrics.py:57-102,137-167` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; production `AsyncLLM` benchmark frontend + auditable scheduler depth; #206 default pre-encodes every prompt before `t0` and admits token IDs, exact `VT_BENCH_PRETOKENIZE=0` retains timed-string admission, and `BenchResult` reports the resolved path (`examples/bench/bench_core.h:190-233,616-665`). Atomic queue/wave admission prepares every request and collector before one all-or-zero ordered core publish (`include/vllm/v1/engine/core_proc.h:90-126`; `src/vllm/v1/engine/async_llm.cpp:125-237`) | server-help/production-frontend/metric assertions plus #206 parser/callback identity, preparation-before-clock, special-token/InputProcessor parity, report-mode capture and synthetic exact-ID A/B (`tests/examples/test_bench.cpp:123-186,347-376`). Atomic queue 3/3·13 (`tests/vllm/v1/test_engine_core_proc.cpp:233-299`); ordered/rollback/shutdown wave gates and complete async suite 12/12·433 (`tests/vllm/v1/test_async_llm.cpp:288-484`), all CPU-GREEN. Real `a33993a7` A/B **FAILED** token identity (98/128 requests, 15,507/16,384 positions); no timing credit. Fresh mutation review, operator gate and real counterbalanced retry remain pending | [CLI/serve/benchmark spike](specs/cli-serve-bench.md); [#206 campaign contract](specs/sm120-qwen35-pareto-2026-08-09.md) | `PARTIAL` | - |
 | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp:265`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ANCHOR-BACKFILL` | CLAIM-SERVE-GATE-1 |
 | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - |
 | `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | Required W1-W11/W13 implementation is complete in draft PR #196: ten-SM gencode/AOT, adaptive CPU tiers, extracted-archive validation and supply chain, least-privilege immutable handoff, eight primary bundles, byte-derived indexes, attestation, and exact-file publication; W12 remains optional/non-primary | Local manifest/archive/workflow/mutation gates, full adaptive x86 tier execution, clean CPU archive, and Vulkan 35/35 backend + 11/11 cross-device archive run are green; hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, and tagged publication remain pending, so no published-binary claim exists | [release-binary-matrix.md](specs/release-binary-matrix.md) | `ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |

.agents/kernel-matrix.md

@@ -172,6 +172,21 @@ and the profiled enclosing workload improves **2.272%**; pinned vLLM remains
 Lifecycle stays `INVENTORIED` because generic Mamba coverage and the 27B/35B
 release gates are unchanged. [Spec and evidence](specs/sm120-qwen35-conv-chunking-2026-08-07.md).
 
+**2026-08-09 `KERNEL-SSM-MAMBA` sm_120 campaign anchor.**
+[#206](https://github.com/mudler/vllm.cpp/issues/206) tracks the RTX 5070 Ti
+Qwen3.5-4B Pareto campaign. Reviewed atomic pretoken admission removed the
+frontend/batching confound; the corrected exact three-repetition comparison is
+**6831.71 vs 6643.40 tok/s (1.0283x)**, while TTFT, TPOT/ITL and E2E remain
+**1.0853x / 1.0165x / 1.0288x slower** and VRAM is **+118.7 MiB**. The exact
+default-OFF GDN decode BV16+swizzle+REGSTATE stack improves local throughput,
+TTFT, TPOT and E2E, and slack-only memset adds a smaller further local win.
+Two deeper candidates are closed: geometric argmax scratch merely moved wait
+from `cudaFree` to stream synchronization and regressed TPOT ~1%; BF16 vector
+writeback improved y800 only 0.143% with one losing raw leg. Both products/tests
+were removed while their specs and same-tool traces remain. Lifecycle stays
+`INVENTORIED`; generic Mamba and 27B/35B coverage are unchanged.
+[Campaign spec](specs/sm120-qwen35-pareto-2026-08-09.md).
+
 ## Count invariants
 
 - This table has exactly 35 practical kernel-family rows.