Skip to content

Commit 3459d2e

Browse files
committed
docs(protocol): ground truth = the WHOLE chain (flashinfer/cutlass/cuBLASLt/Inductor), not just vLLM repo
Per user: vLLM is orchestration; the kernels live in its deps. Never declare a lever build-specific/unverifiable without inspecting the dep chain + dumping generated code. PROOF: flashinfer/cute_dsl/add_rmsnorm_fp4quant.py IS the fused rms+fp4-quant kernel we wrongly declared nonexistent from vLLM csrc alone. Adds .agents/parity-lever-protocol 'Verify the whole chain' + broadens AGENTS.md GROUND-EVERY-CHECK. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Claude-Session: https://claude.ai/code/session_01UJyFKcK62CcR3imhgbiBnW
1 parent ac46564 commit 3459d2e

2 files changed

Lines changed: 52 additions & 7 deletions

File tree

.agents/parity-lever-protocol.md

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,9 +76,39 @@ In one session the loop turned every "ceiling" into a concrete lever and took th
7676
dense at 72ms when it was 15ms, and named two neutral levers).
7777
- vLLM's baselines **drift** — always re-measure the vLLM denominator on the
7878
IDENTICAL workload (e.g. 35B 2768→3145, 27B 397→452 across re-measures).
79-
- Distinguish the few vLLM edges that are **build-specific** (Inductor epilogue
80-
fusion, DeepGEMM, flashinfer-cutlass exact accumulation) — which eager-C++
81-
cannot replicate 1:1 — from the many that are plain code/config diffs we CAN.
8279
- Compare at the **right operating point**: "large concurrency" is the gate; a
8380
single-wave (np=conc) measurement hides sustained-load effects, and a
8481
memory-tight model may need a different concurrency than the 35B.
82+
83+
## Verify the WHOLE chain (vLLM is orchestration; the kernels live in its deps)
84+
85+
**Do not scope "what vLLM does" to the vLLM repo.** vLLM dispatches to kernels in
86+
its dependencies; a lever "absent from vLLM's `csrc/`" often lives one layer down.
87+
Before calling anything "build-specific / unverifiable / out of reach", inspect the
88+
chain and, for compiled/JIT code, DUMP the generated kernel:
89+
90+
- **flashinfer** (`~/venvs/vllm-oracle/lib/python3.12/site-packages/flashinfer/`):
91+
the real fp4/fp8 GEMMs + fused kernels, many as **readable CuTe-DSL Python** — e.g.
92+
`cute_dsl/add_rmsnorm_fp4quant.py` (fused Add+RMSNorm+FP4-quant), `cute_dsl/
93+
rmsnorm_fp4quant.py`, `gemm/` (the fp4 GEMM + its tactic selection),
94+
`fused_moe/cute_dsl/blackwell_sm12x/` (sm_121-specific). CuTe-DSL is portable — read
95+
it and hand-write the equivalent CUDA. (This is how we found the rms+fp4 fusion we
96+
had WRONGLY declared nonexistent from vLLM's csrc alone — 2026-07-07.)
97+
- **cuBLASLt kernel selection** (closed, but OBSERVABLE): `nsys` shows the exact
98+
kernel name (e.g. `cutlass_80_tensorop_*` sm_80 vs `nvjet_sm121_*`); probe
99+
`cublasLtMatmulAlgoGetHeuristic` to see what it picks for a given shape/dtype
100+
(e.g. bf16→f32 vs bf16→bf16 output) before assuming a reselect.
101+
- **torch/Inductor**: run the real oracle with `TORCH_LOGS=output_code` /
102+
`TORCH_COMPILE_DEBUG=1` and READ the generated Triton — the fusions are a specific,
103+
finite set for these specific ops, not "arbitrary". Most are hand-portable once read.
104+
- **cutlass / DeepGEMM**: vendored + in flashinfer; read the actual tile/config/
105+
scheduler the dep instantiates for our shapes and mirror it.
106+
107+
Only after reading the dep source AND (for JIT/compiled) dumping the generated kernel
108+
may you conclude a lever is genuinely irreducible for eager-C++ — and say exactly why
109+
(cite the dep file:line / the dumped kernel). The default assumption is PORTABLE.
110+
111+
- Distinguish the few genuinely irreducible edges (proven so by the above) from the
112+
many that are plain code/config/dtype/algorithm diffs — but prove it, don't assume
113+
it. The CLEAN nsys slice of OURS (above) tells you WHERE to look; the dep chain
114+
tells you WHAT vLLM actually runs there.

AGENTS.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,25 @@ vLLM's selection logic, including what it selects on GB10/sm_121. Only escalate
3838
genuine PRODUCT/scope calls vLLM can't answer (e.g. "is model X in the MVP?"),
3939
never "how should feature X behave?" (→ mirror vLLM).
4040

41-
**GROUND EVERY CHECK IN vLLM SOURCE.** Do not decide behavior from memory or
42-
assumption — read the actual pinned vLLM code (`/home/mudler/_git/vllm` @ the
43-
parity pin `e24d1b24`) every time, cite `file:line`, and mirror what you find.
44-
This applies to every subagent and every design/parity check.
41+
**GROUND EVERY CHECK IN THE WHOLE EXECUTION CHAIN, not just the vLLM repo.** vLLM
42+
is an ORCHESTRATION layer — the kernels that actually run (and that make it fast)
43+
live in its DEPENDENCIES: **flashinfer** (CuTe-DSL / cutlass fp4·fp8 GEMMs, fused
44+
norm+quant, MoE, sm_121 "blackwell_sm12x" kernels), **cutlass**, **cuBLASLt**
45+
(nvjet), **DeepGEMM**, and **torch/Inductor** (the fused Triton it codegens). Read
46+
the actual pinned vLLM code (`/home/mudler/_git/vllm` @ pin `e24d1b24`) AND, as
47+
needed, the installed dep source (`~/venvs/vllm-oracle/lib/python3.12/site-packages/`
48+
— e.g. `flashinfer/cute_dsl/*.py`, `flashinfer/gemm/`), cite `file:line` on every
49+
side, and mirror what you find. **NEVER declare a lever "build-specific",
50+
"unverifiable", or "out of reach" without first inspecting the dep chain and, for
51+
compiled/JIT/Inductor code, DUMPING the generated kernel**`TORCH_LOGS=output_code`
52+
/ `TORCH_COMPILE_DEBUG=1` for Inductor Triton; nsys kernel names +
53+
`cublasLtMatmulAlgoGetHeuristic` for cuBLASLt selection; the CuTe-DSL / cutlass
54+
source for flashinfer. A fusion "absent from vLLM's csrc" may live in flashinfer
55+
(e.g. `add_rmsnorm_fp4quant` — the fused Add+RMSNorm+FP4-quant we initially and
56+
WRONGLY declared nonexistent). Verify the whole chain as necessary. This applies to
57+
every subagent and every design/parity check. Full method:
58+
[.agents/parity-lever-protocol.md](.agents/parity-lever-protocol.md) § Verify the
59+
whole chain.
4560

4661
## STANDING DIRECTIVE — always compare vs vLLM (the oracle), same workload
4762

0 commit comments

Comments
 (0)