Skip to content

perf(gdn): optimize post-conv and causal-conv kernels - #155

Open
richiejp wants to merge 10 commits into
mudler:mainfrom
richiejp:row/KERNEL-SSM-MAMBA-POSTCONV-TILE
Open

perf(gdn): optimize post-conv and causal-conv kernels#155
richiejp wants to merge 10 commits into
mudler:mainfrom
richiejp:row/KERNEL-SSM-MAMBA-POSTCONV-TILE

Conversation

@richiejp

@richiejp richiejp commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Row

KERNEL-SSM-MAMBA — Qwen GDN post-conv and causal-conv kernel sublevers; one row per PR.

Before starting

  • Issue/PR search and existing claim: this is a stacked follow-on to perf(gdn): dispatch exact causal-conv chunks #127, which supplies the exact causal-conv chunk descriptors measured here. No separate matching open PR was found for these two opt-in kernel specializations.
  • Roadmap or matrix row, plus scripts/ready-for-helper.py result when applicable: .agents/kernel-matrix.md:157 (KERNEL-SSM-MAMBA, INVENTORIED), feeding ROAD-V1-C2-LOCAL-BF16. Spikes: .agents/specs/sm120-qwen35-postconv-token-tile-2026-08-08.md and .agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md.
  • Exact current-code and test/evidence anchors inspected: src/vt/cuda/cuda_gdn.cu, src/vt/cuda/gdn_prefill_conv.h, tests/vt/test_gdn_prefill_conv.cpp, tests/vt/test_ops_gdn.cpp, and pinned vLLM/FLA causal-conv plus fused post-conv kernels recorded with file:line anchors in the two spikes.

What changed

Add two independently selectable, default-off CUDA specializations. VT_GDN_POSTCONV_TOKEN_TILE=1 ports the upstream-shaped 16-token/four-warp post-conv schedule while preserving the existing 128-lane reduction tree byte-for-byte. VT_CONV_CHANNEL_TILE=1 specializes the causal-conv path for width four at the existing one-channel-per-thread grid; arm =2 is retained as the measured 256-channel/two-channels-per-thread falsification arm. Production dispatch and the portable mutation gate share one callback dispatcher, so the tests prove that each selected arm is actually invoked.

Evidence

  • scripts/agent-preflight.sh passes after rebasing onto canonical upstream/main c05cee1d2; all record gates, mutation suites, live-row audit, and committed-range document checks are green.
  • Tests covering this change: contained CUDA rebuild including an explicit vllm-bench relink; portable dispatch 9/9 cases, 88/88 assertions; CUDA GDN 67/67 cases, 4631/4631 assertions; Qwen3.5 paged-forward 4/4 cases, 8/8 assertions. The accepted production arms produce identical token hashes.
  • Same-change doc obligations: docs/STATUS.md and docs/BENCHMARKS.md updated; the model/feature surface did not expand. Full profiler evidence and the void stale-link series are retained in the two specs and .agents/benchmark-record.md.

Speed claims

  • The operator ran the numbers under ${GPU_LOCK} and they are recorded in docs/BENCHMARKS.md with the repro recipe.

On the local RTX 5070 Ti / Qwen3.5-4B c32 workload, the post-conv tile reduces graph-node time 227.887 to 122.587 ms (1.859x) and improves every observed enclosing axis, including total/output throughput by 0.532%. The causal-conv K4 arm reduces graph-node time 234.605 to 219.506 ms (6.44%) and improves whole-run total/output throughput by about 0.12%. The 256-channel arm is 4.05% slower than the K4 arm and is recorded as falsified.

Honest gaps

  • This PR is stacked on perf(gdn): dispatch exact causal-conv chunks #127. Until perf(gdn): dispatch exact causal-conv chunks #127 merges, GitHub shows its exact-chunks commits in this PR too; after that merge, this PR reduces to the opt-in follow-on commits.
  • Both specializations remain opt-in pending repeated A/B and hardware-unavailable 27B/35B gates. The 4B result is not extrapolated to the release models.
  • Same-tool residuals remain: post-conv is 1.135x and causal conv K4 is 1.509x slower than pinned vLLM.
  • The first production causal-conv profile series was void because vllm-bench had not relinked the rebuilt CUDA library. The accepted series explicitly relinked the benchmark and trace-proved the specialized kernels.
  • Current main c05cee1d2 independently fails check-device-leakage.py: src/vllm/v1/worker/gpu/runner.cpp contains one new hard-coded DeviceType::kCUDA (DSR kcuda 1 > baseline 0). perf(gdn): dispatch exact causal-conv chunks #127 and this PR inherit that merged-tree failure; neither kernel diff introduced it.

richiejp added 10 commits August 8, 2026 14:23
Build and upload exact causal-convolution work descriptors once per step, reuse them across GDN layers, and make the exact register-kernel mapping the default. Keep the production AsyncLLM benchmark path in the same checkpoint so the measured scheduler and kernel improvement is reproducible on current upstream/main.

Row: row/KERNEL-SSM-MAMBA-EXACT-CHUNKS

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
Canonicalize semantic consumer bodies without Python-version-only AST fields, and gate the representation under Python 3.12 and 3.13.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
Make the row-PR provenance visible to GitHub's synthetic merge check without changing the measured tree.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
Record the merged Gemma-4 MoE path as known merged-GEMM drift and close the stale embeddings claim after PR mudler#137 landed.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
Select the measured sm_120 post-conv residual, reject the existing split grid, and bind the upstream 16-token CUDA experiment and gates.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
Port the upstream 16-token per-head schedule behind an opt-in flag, preserve the existing reduction order byte-exactly, and record the sm_120 A/B evidence.

Row: row/KERNEL-SSM-MAMBA-POSTCONV-TILE

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Codex:gpt-5.6-sol [Codex]
Route runtime-width and both K=4 CUDA launch arms through the same portable, mutation-tested callback dispatcher. Pin strict arm-2 parsing and resolved non-K4 fallback semantics.

row/KERNEL-SSM-MAMBA-POSTCONV-TILE

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Codex:gpt-5.6-sol [Codex]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant