Skip to content

perf(gdn): optimize post-conv and causal-conv kernels #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
richiejp wants to merge 10 commits into
base: main
Choose a base branch
from
Open

perf(gdn): optimize post-conv and causal-conv kernels #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
39da53c
perf(gdn): dispatch exact causal-conv chunks
richiejp Aug 7, 2026
b50f53e
fix(protocol): stabilize release AST digests
richiejp Aug 8, 2026
4d0538b
record(pr): bind row/KERNEL-SSM-MAMBA-EXACT-CHUNKS #127
richiejp Aug 8, 2026
32eb2f0
fix(protocol): repair current-main lifecycle gates
richiejp Aug 8, 2026
fd64da4
docs(gdn): spike post-conv token tiling
richiejp Aug 8, 2026
c96cadf
perf(gdn): tile fused post-conv tokens
richiejp Aug 8, 2026
765aeff
docs(gdn): spike causal-conv channel tiling
richiejp Aug 8, 2026
e224553
perf(gdn): add causal-conv channel tile arms
richiejp Aug 8, 2026
680dff7
test(gdn): bind causal-conv arm dispatch
richiejp Aug 8, 2026
ffbc821
record(gdn): select K4 causal-conv specialization
richiejp Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions .agents/NOW.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,7 @@ benchmark record. Budget: 100 lines.

## Live claims

Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
`bench/qwen35-upstream-rebenchmark-20260805` on `upstream/main` @ `59674cf1d`.
Work: #127 rebased on main `c05cee1d2`; opt-in post-conv and K4 conv follow-ons.

| Claim / track | State | Next command or step |
|---|---|---|
Expand All @@ -20,7 +19,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
| MiniMax-H3 lane | **bf16 shards STREAM both towers (DiT + encoder); Q4_K_M enc cond cos 0.9975, 3.5° med, DIFFUSE** | render A/B on saved embeds |
| Kimi-Linear-48B | **ROW 7 fold LANDS (#122 §21): engine==CLI 128/128; golden 122/128; SACRED green; v13 tokens ABI** | ACTIVE: 19.0 tok/s vs vLLM ~21 (~0.90×) |
| 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
| Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
| Qwen3.5-4B sm_120 | Exact chunks ON 3.072x. Post-conv tile 1.859x. K4 conv arm 1 opt-in: exact, 6.44%; 256-channel arm falsified | Repeat A/B; 27B/35B gates unavailable |
| RPi5 A76 CPU | **R5 asm GREEN; llama NOT MET**: 0.461x pf, 0.653x dec, RSS -24% | W6: BF16 GEMM |
| MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity |
| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner |
Expand All @@ -30,7 +29,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
| `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed (contributor, authorship-preserved); W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) |
| TP spike #287 (PR #143) | **LANDED** ([spec](specs/tensor-parallelism-spike.md)); DSpark rider grounded | dispatch TP-W1 (CPU-able) |
| Release | SPIKE; 30/30 | #129 |
| Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139 IN; **ROW 6 IN REVIEW (#137): embeddings LIVE — `LlamaModel` arch, PoolingRunner in the step, `vllm_embed` v15, `/v1/embeddings`, fold gate 4/4-231, 9 kills** | Merge #137; real-ckpt oracle cosine residual |
| Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139 IN; **ROW 6 LANDED (#137): embeddings LIVE — `LlamaModel` arch, PoolingRunner in the step, `vllm_embed` v15, `/v1/embeddings`, fold gate 4/4-231, 9 kills** | Real-checkpoint oracle cosine residual |

In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna
bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.
Expand All @@ -46,18 +45,19 @@ both gate models, reproduced 2–3x on an idle box. See [gates.md](gates.md) and

## Next actions

0. **`ROAD-V1-MEM`** KV auto-sizing spike LANDED (`specs/kv-sizing.md`, `READY`).
0. **`ROAD-V1-MEM` M1+M2 LANDED:** `--kv-cache-memory` + group-aware divisor;
M3 profile run is dgx-gated.
1. **Spike the Parakeet encoder row** (vLLM carries it inside
`nano_nemotron_vl.py`; the transducer half is NOT in vLLM: separate call).
2. **Qwen3.5-4B serving follow-up:** bind the default-ON async-serving path
against the same oracle before attributing the remaining TPOT gap.
2. **Merge the invocation-parity prevention** (CI guard + AGENTS.md checklist);
2. **Qwen3.5-4B sm_120:** post-conv and K4 conv are exact/local-positive but
opt-in; repeat A/B or divide the remaining causal-conv residual further.
3. **Merge the invocation-parity prevention** (CI guard + AGENTS.md checklist);
CUDA build-verify the byte-exact `kGemvHeuristicAlgos` refactor on dgx.
3. **Same-tool re-verify deepseek_v4's bf16 resident tower** (the one other
4. **Same-tool re-verify deepseek_v4's bf16 resident tower** (the one other
f32-out caller) once the Laguna fix proves the mechanism.
4. **Restore `local-ai-worker`** on dgx when the GPU campaign ends
5. **Restore `local-ai-worker`** on dgx when the GPU campaign ends
(`docker update --restart=always` + `docker start`).
5. **Protocol substrate — partly done.** Triage/audit + `STATUS.md` ratchet +
6. **Protocol substrate — partly done.** Triage/audit + `STATUS.md` ratchet +
`AGENTS.md` tiering DONE. REMAINING: anchor backfill (6 model rows need a
DECISION); record-era rollover BLOCKED on `DONE` rows bound to
`parity-ledger.md` LINE anchors (re-anchor by ROW ID).
Expand All @@ -82,7 +82,6 @@ rows**; llama.cpp's 11 extra devices IN SCOPE, spike-gated (`ROAD-V1-D6`).
- Mirror vLLM; never ask how a feature should behave.
- `nsys` BOTH sides, SAME tool, before any perf claim; cross-tool comparisons
never establish invocation parity; whole-run sums mix prefill.
- GPU: park `local-ai-worker`, flock `$HOME/gpu.lock`, single-load
steady-state, never reload per rep, named tmux.
- GPU: flock `/tmp/gpu`; use the 22/25 GiB user-systemd scope; single-load reps.
- Never weaken a checker to pass; repair the record.
- Feature code needs a `row/*` PR (enforced); integration paths push direct.
190 changes: 188 additions & 2 deletions .agents/benchmark-record.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15124,8 +15124,6 @@ llama.cpp's Vulkan" when it is really "our CPU tier vs llama.cpp's Vulkan". The
comparison becomes meaningful when native coverage closes — the progress metric is
`vt::GetReferenceTierHits()` reaching 0, and the ops that matter for this model are
the RoPE table build, the sampler tail, and the remaining norm/glue set.
>>>>>>> 814230a0 (bench(vulkan): VK-E unblocked with identical weights; ours quoted as NO RATIO)

#### CORRECTION (2026-08-07, same session): the vllm.cpp Vulkan arm is GPU-BOUND, not CPU-bound

The entry above attributes vllm.cpp's slowness to "our HOST FALLBACK wearing a
Expand Down Expand Up @@ -15983,3 +15981,191 @@ loads, or caching the K/V slice across the query heads that share a KV head —
Recorded because the hypothesis was specific and the refutation is reusable: this
is the second kernel this session where a barrier-count argument looked compelling
and measured flat (the subgroup GEMV was the first).

## 2026-08-07 — sm_120 exact GDN causal-conv chunks: 3.069x kernel, +2.152% enclosing

**Disposition:** ACCEPTED and reproduced on clean current-main transplant
`upstream/main` `f91a5917a`. Exact chunks default ON; latency, VRAM and
27B/35B gates stay open.

The metadata builder enumerates exact `(sequence, 8-token chunk)` work, uploads
its two i32 descriptors once per step and shares them across GDN layers. The
register kernel consumes one descriptor per `grid.y`.
`VT_CONV_EXACT_CHUNKS=0` is the same-binary whole-sequence rollback;
`VT_CONV_REG=0` selects tiled/scalar.

**Correctness.** RED compile evidence:
`/tmp/vllm-agent-runs/gdn-exact-red-escalated.json`. Focused host metadata and
flags, affected Qwen fixtures, full CUDA GDN and cached Qwen3.5-4B 3/3·1672
were green. Three production ON/OFF pairs had identical token files for all
128 requests × 128 outputs.

**Same-binary profile.** Manifest
`/tmp/vllm-agent-runs/qwen35-conv-exact-ab-profile.json`; rollback/default
traces `/tmp/qwen35-conv-exact-{off,on}.nsys-rep`. Rollback: 1728 calls,
720.047171 ms, 416.694 us mean. Exact: 1728 calls, 234.607112 ms, 135.768 us.
That is **3.069x**, saving 485.440 ms. Exact `grid.y=279/280/282` work counts
replace dominant `(64,28..32,1)` whole-sequence grids, confirming the proposed
mechanism. Pinned-vLLM same-tool total is 145.421 ms; residual **1.613x**.

**Enclosing A/B.** Manifest
`/tmp/vllm-agent-runs/qwen35-conv-exact-local-ab.json`; evidence root
`/tmp/qwen35-conv-exact-local-ab-20260807`. Three alternating pairs under one
GPU lock and a 25 GiB user-systemd scope:

| Axis | rollback | exact default | ratio |
|---|---:|---:|---:|
| total | 6641.800 tok/s | 6784.743 tok/s | 1.02152x |
| output | 734.433 tok/s | 750.237 tok/s | 1.02152x |
| TTFT | 1048.927 ms | 1018.040 ms | 0.97055x |
| TPOT / ITL | 35.420 ms | 34.740 ms | 0.98080x |
| E2E | 5547.687 ms | 5430.180 ms | 0.97882x |

Against sealed vLLM, exact local is **1.021246x** throughput,
**1.085812x** TTFT and **1.024597x** TPOT. Exact peak VRAM
13044/13058/13058 MiB, mean 13053.3, versus old local 13054 and vLLM 12820.

**VOID oracle attempts.** `qwen35-conv-exact-default-full-compare.json` exposed
the missing live-driver link path; the harness now adds `/run/opengl-driver/lib`
to `LIBRARY_PATH`. `qwen35-conv-exact-default-full-compare-rerun.json` reached
13/18 legs before a transient Torch bytecode read invalidated Triton AOT cache
keys. Neither attempt supersedes the sealed denominator. Full evidence:
`docs/bench-evidence/qwen35-4b-sm120-main-20260807.md`.

**Clean-transplant reproduction (`f91a5917a`).** Contained CPU/CUDA rebuild;
focused CPU 6/6, full CUDA GDN 66/66·4300, cached 4B 3/3·1672. Same-binary
graph-node traces `/tmp/qwen35-conv-exact-transplant-{off,on}.nsys-rep` and
token files reproduce the mechanism with byte identity: rollback 1728 calls /
720.216507 ms / 416.792 us, exact 1728 / 234.379395 ms / 135.636 us =
**3.072866x**. Profiled enclosing totals are 6587.66→6727.35 tok/s
(**1.021205x**), TTFT 1058.73→1025.46 ms, TPOT 35.70→35.04 ms and E2E
5592.69→5475.91 ms. The transplanted result is therefore reproduced, not merely
carried from its old branch. Against the sealed vLLM conv trace, residual is
**1.611730x**.

**Post-rebase reproduction (`3d2581551` on `upstream/main` `48a54141f`).** The
contained rebuild and all three gates remain green: focused 6/6, CUDA GDN
66/66·4300, cached 4B 3/3·1672. Fresh graph-node traces and token files under
`/tmp/qwen35-conv-exact-rebase-3d2581551-{off,on}.*` are byte-identical.
Rollback is 1728 calls / 718.704016 ms / 415.917 us; exact is 1728 /
233.954533 ms / 135.390 us = **3.07198x**. Profiled enclosing totals are
6589.65→6739.34 tok/s (**1.02272x**), TTFT 1057.63→1022.70 ms, TPOT
35.70→34.99 ms and E2E 5590.92→5466.20 ms. The result therefore survives the
27-commit main advance; against the sealed vLLM conv trace the residual is
**1.60881x**. Trace SHA-256: rollback `6a5dde18e...f97c47`, exact
`f47fb9cc...7aecf9`; both token files `83fcdc45...453545`.

## 2026-08-08 — sm_120 fused GDN post-conv 16-token tile: 1.859x kernel, byte-exact

**Disposition:** IMPLEMENTED as opt-in `VT_GDN_POSTCONV_TOKEN_TILE=1`.
Locally positive and token-safe; default and release-model gates remain open.

**Selection and falsification.** On exact-chunks `c3bb0f39a`, the accepted
Qwen3.5-4B/c32/1,280-block graph-node trace measured fast megablock
228.150171 ms and the existing per-V-head split 448.364941 ms across 1,728
calls. Both token files were identical. The split is 1.965x slower and the
enclosing run 0.97% slower, so V-only launch decomposition is rejected.
Pinned vLLM's same-tool `_fused_post_conv_kernel` is 108.034870 ms across 1,923
calls. Source comparison instead selected vLLM/FLA's 16-token, per-head,
four-warp schedule and Q/K register reuse.

**Implementation and numerical mutant.** `GdnPostConvTokenTileKernel` maps one
block to `(16 tokens, one Q/K or V head)` and one warp to four tokens. Each Q/K
lane retains features `lane+{0,32,64,96}` across normalization. The first
implementation summed those four squares sequentially before a warp reduction:
kernel time improved 227.731960→122.472980 ms and enclosing throughput
6731.69→6773.85 tok/s, but the tile token SHA
`1d496ff0f989978155d8e900c7a5500a43db26816dead8e035310d0bf9cb9756`
did not match fast `83fcdc45...453545`; REJECTED. Reproducing the current
128-lane tree exactly—`(i+i+64)`, then `(i+i+32)`, then shuffle offsets
16/8/4/2/1—restored byte identity without restoring the reload/barrier costs.

**Final same-binary profile.** One `/tmp/gpu` lock, 22/25 GiB user-systemd
scope, `--cuda-graph-trace=node`, identical production workload and binary:

| Axis | fast megablock | token tile | change |
|---|---:|---:|---:|
| post-conv GPU total, 1,728 calls | 227.887066 ms | **122.587027 ms** | **1.858982x faster** |
| mean post-conv call | 131.879 us | **70.942 us** | **46.21% lower** |
| total throughput | 6734.82 tok/s | **6770.62 tok/s** | **+0.532%** |
| output throughput | 744.72 tok/s | **748.68 tok/s** | **+0.532%** |
| TTFT | 1024.14 ms | **1015.43 ms** | **-0.850%** |
| TPOT / ITL | 35.01 ms | **34.85 ms** | **-0.457%** |
| E2E | 5469.87 ms | **5440.81 ms** | **-0.531%** |

The vLLM kernel residual is now **1.134699x**, down from 2.112x. Final token
files are identical, full SHA-256
`83fcdc45f79ddb06a634c7d7d95eba3384543b3cd781a45a8db1fc4e2a453545`.
Portable flag/grid tests pass 6/6·50; CUDA GDN passes 67/67·4384 including
partial/exact tiles, packed BA non-zero views/wider row strides, byte-exact five
outputs, finiteness and norms; cached Qwen3.5-4B passes 3/3·1672.

**Evidence.** Final fast/tile traces
`/tmp/qwen35-postconv-tile-exact-{fast,tile}.nsys-rep`, SHA-256
`c75e2cb...bcadc` / `a0eb1808...f418`; token files beside them. Rejected
arithmetic traces `/tmp/qwen35-postconv-tile-wip-{fast,tile}.nsys-rep`, SHA-256
`c2d8872e...b774` / `1c42d489...d66b`. Selection traces and exact recipes are
in [the spike/result](specs/sm120-qwen35-postconv-token-tile-2026-08-08.md).
One local profile is not extrapolated to the unavailable Qwen3.6-27B/35B gates;
the flag therefore remains opt-in.

## 2026-08-08 — sm_120 causal-conv residual: K=4 specialization wins; 256-channel tile falsified

**Disposition:** arm 1 is IMPLEMENTED, byte-exact and locally positive behind
`VT_CONV_CHANNEL_TILE=1`; arm 2 (`=2`) is retained as an explicit falsified
experiment. Unset/`0` remains the default pending repeated and release-model
gates.

**Divide-and-conquer selection.** Fresh current-main/post-conv tracing measured
causal conv 234.255 ms versus pinned-vLLM 145.532 ms, while post-conv's
remaining excess was only 14.476 ms. Grouping causal-conv launches by grid
showed the 279/280-program waves consumed 136.189 ms (58.1%). Local used 64
feature blocks and runtime width at 43 registers/thread; vLLM used `BLOCK_N=256`,
32 feature blocks and compile-time width at 32 registers/thread. The spike split
those differences into arm 1 (compile-time K=4, unchanged 64 blocks) and arm 2
(K=4 plus two channels/thread, 32 blocks).

**Correctness/review finding.** A serial-stripe arm 2 corrupted the second
stripe when an exact final-chunk block wrote state before stripe 2 loaded initial
history; the CUDA matrix caught it and the accepted kernel preloads both stripes.
Fresh mutation review then found byte comparisons could stay green when a whole
specialized dispatch branch was deleted. A fresh fix routes production through
the same portable callback dispatcher the tests mutate. Scoped re-review killed
arm-1 deletion, arm-2 deletion, relaxed arm-2 parsing and removed non-K4 fallback.
Final gates: portable 9/9·88, CUDA GDN 67/67·4631, paged-forward 4/4·8.

**VOID series.** `/tmp/qwen35-conv-arm{0,1,2}-565a26fcc.*` is invalid for
selection: the test executables had rebuilt but `vllm-bench` had not relinked.
All three traces proved the old runtime kernel/grid/registers ran. This was
caught structurally before timing interpretation.

**Accepted rebuilt same-binary profile.** One GPU lock, 22/25 GiB user-systemd
scope, exact c32 workload and `--cuda-graph-trace=node`:

| Axis | arm 0 runtime | arm 1 K4x1 | arm 2 K4x2 |
|---|---:|---:|---:|
| kernel / grid / registers | runtime / 64 / 43 | K4<1> / 64 / 52 | K4<2> / 32 / 58 |
| causal-conv total, 1,728 calls | 234.604587 ms | **219.506425 ms** | 228.400830 ms |
| 279-program mean | 149.546 us | **140.133 us** | 145.586 us |
| 280-program mean | 149.480 us | **139.982 us** | 145.468 us |
| total throughput | 6759.39 tok/s | **6767.62 tok/s** | 6757.19 tok/s |
| output throughput | 747.43 tok/s | **748.34 tok/s** | 747.19 tok/s |
| TTFT | 1016.69 ms | **1013.82 ms** | 1017.61 ms |
| TPOT / ITL | 34.91 ms | **34.88 ms** | 34.91 ms |
| E2E | 5449.87 ms | **5443.16 ms** | 5451.66 ms |

Arm 1 improves conv **6.4356%**, with every enclosing axis positive but small.
It leaves a **1.5083x** same-tool vLLM conv residual. Register count rises rather
than falls, falsifying the occupancy rationale; compile-time removal of runtime
width work is the supported cause. Arm 2 improves only 2.6443% versus baseline,
is **4.0520% slower than arm 1**, and is neutral/slightly negative end to end:
halving blocks does not repay duplicated channel-local register state on sm_120.

All accepted token files SHA-256
`83fcdc45f79ddb06a634c7d7d95eba3384543b3cd781a45a8db1fc4e2a453545`.
Report SHA-256 arm0/arm1/arm2:
`39d383dd878fc340a3cfaaee79a4addcb4eccb181439e9b4725f724f4569a6eb`,
`c8799ac0b4cdf997d383fe8a690b223be882dce3b1ee1a6fff35a62d75f7cf85`,
`3d38793571539864b23688fd9a85966debbf1e7c48fe8a1a2509438a45ee0452`.
Full recipe and decision:
[structured spike/result](specs/sm120-qwen35-conv-channel-tile-2026-08-08.md).
Loading
Loading