ledger: conc-32 is idle-bound (host blocked in memcpy 98% API time, GPU 58% busy); flat scaling; vLLM nsys uncooperative; async-scheduling lever under investigation

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Claude-Session: https://claude.ai/code/session_01UJyFKcK62CcR3imhgbiBnW

Author: mudler

.agents/parity-ledger.md

@@ -236,3 +236,5 @@ Columns:
 | 2026-07-08 (conc-32 gap ROOT-CAUSED: host↔GPU memcpy SYNC, not kernels — batch-cap + mnbt hypotheses DISPROVEN) | Chased the conc-32 scaling gap (0.785×). (1) BENCH METHODOLOGY: our bench caps batch at `concurrency` (max_num_seqs=conc + client in-flight cap, bench_core.h:396/426) while vLLM `bench throughput` runs the full np=192 backlog at max_num_seqs=1024. HYPOTHESIS: unfair. DISPROVEN by the fair test below. (2) FAIR matched-batch (in1024/out128 np192, both batch≤32, GM0.6): ours=829.81, vLLM **graphed** max_num_seqs=32 = **1051.53 → 0.789×** (≈ the 'unfair' 0.785×); vLLM graphed batch-32 ≈ batch-192 (~1055) because the workload is PREFILL-BOUND (89% prefill) so batch size barely matters. So the harness cap was NOT the cause. Decode-vs-eager: ours 829 / eager-vLLM 862 = 0.96×; eager→graphed = +22% (vLLM's torch.compile fusion + async/graph). (3) mnbt (prefill budget) A/B at conc-32: default(8192)=829, 16384=788, 24576=803 → NEUTRAL (Explore's 'raise to 16384' DISPROVEN; the 454 for explicit-8192 was a measurement glitch — same config as default). | (measurement + 2 Explores + nsys) | **STEADY-STATE nsys (load EXCLUDED, window t=60-110s of an np256 run): `cudaMemcpyAsync` = 98.8% of API time (49s of a 50s window), 23,872 calls, median 7.5us but MAX 168ms → a few hundred BLOCKING stalls; GPU only ~58% busy on kernels (cutlass GEMM 43%, nvjet 8%, GDN chunk WMMA ~16%), ~42% IDLE on host memcpy sync. cudaGraphLaunch=122 (decode IS graphed) yet host stalls → sync is OUTSIDE the graph. CODE (Explore a4a91f8): the DECODE path is already fully hoisted/on-device (6-7 small H2D/step, GDN conv+ssm state in-place with gidx uploaded once, sampler downloads only token-ids); BUT the PREFILL GDN path (`GdnBlockPaged` np>0) STILL rebuilds step-constant metadata (dqsl/dhis/dpqsl from `meta.*`) as fresh pageable H2D uploads INSIDE the per-layer loop (~90 H2D/prefill-step across 30 GDN layers) — each pageable cudaMemcpyAsync blocks until the prior layer drains, serializing the 30 GDN layers per prefill step. Same class of bug the code already fixed for decode (StepDevInputs, 'replaced ~110 copies/step') + GdnPrefill qsl ('~67% GPU-idle in prefill'). LEVER IN PROGRESS: hoist prefill GDN metadata into StepDevInputs (build once/step, reuse across layers) behind VT_HOIST_GDN_PREFILL_META — worktree agent. If gap persists, next lever is the decode token-readback lockstep (async scheduling / on-device input build). KERNEL levers (GDN-as-cutlass-GEMM ~16%, flash_fwd, nvjet re-steer) are BLOCKED behind closing the ~42% host-sync idle first.** |
 
 | 2026-07-08 (conc-32 host-copy fixes — 3 NEUTRAL: prefill-meta-hoist, pin-GDN-state, mnbt; PROTOCOL MISS: never nsys'd vLLM) | Chased the conc-32 host-memcpy idle with 3 targeted, grounded fixes, all behind toggles, all gate 9/9: (1) VT_HOIST_GDN_PREFILL_META (hoist ~87 per-layer prefill metadata H2D/step → once/step) = NEUTRAL. (2) VT_PIN_GDN_STATE (cudaHostRegister the GDN state cache so the prefill gather/scatter cudaMemcpyAsync is async not pageable-sync) = NEUTRAL. (3) mnbt 8192→16384/24576 = NEUTRAL. | gates 9/9 all | **The nsys of OURS showed cudaMemcpyAsync=98.8% of API time + ~42% GPU idle, but making those copies async (pin) did NOT help → the pageable-sync copies are NOT the actionable bottleneck. LESSON/PROTOCOL MISS: I traced ONLY ours and inferred the gap; never nsys'd vLLM at conc-32 (violates 'nsys BOTH sides FIRST'). The 42% idle may be inherent prefill/decode bubbling vLLM ALSO has (→ gap is kernel GPU-busy speed: our cutlass GEMM 43% + GDN chunk WMMA 16% vs vLLM's fused/flash_fwd/GDN-as-cutlass-GEMM) OR real recoverable idle. Decomposition: ours 829 = 0.96× eager-vLLM (862), eager→graphed +22% = vLLM's Inductor prefill FUSION + cudagraph (the KNOWN-hard gap per honest-bar-is-graphed; my rms+fp4 and silu+quant fusion attempts were already neutral). All 3 branches SHELVED. NEXT: nsys graphed vLLM conc-32 (steady-state) to compare GPU-busy% — redirect to kernels (rank 2/3 GDN-as-GEMM + flash_fwd) if vLLM is compute-bound too.** |
+
+| 2026-07-08 (conc-32: IDLE-BOUND not compute-bound; flat scaling; vLLM nsys uncooperative → investigating async scheduling) | Follow-up to the 3 neutral host-copy fixes. (1) SCALING: ours conc-32=826.7 → conc-48=831.7 = FLAT (does not climb with batch). (2) vLLM nsys FAILED both multiprocess AND in-process (VLLM_ENABLE_V1_MULTIPROCESSING=0) — nsys breaks vLLM's EngineCore init (model_executor), so the definitive 'both sides' GPU-busy% is unavailable on this box. | (measurement) | **RE-READ of OUR API profile: cudaMemcpyAsync = 98.8% of API time (49s of a 50s window) = the HOST THREAD is blocked in memcpy ~98% of the time while the GPU is only ~58% busy → GPU idles ~42% WAITING FOR THE HOST. So conc-32 is HOST-BLOCKED / IDLE-BOUND, not kernel-compute-bound. Pinning the STATE cache was neutral because the blocking copies are the PER-STEP sampled-token D2H → next-input H2D round-trip (the decode LOCKSTEP), NOT the GDN state gather/scatter. IMPLICATION: speeding up kernels (FA-2, GDN-as-GEMM) will NOT help while idle-bound — it just grows idle. The idle-reducing lever = vLLM V1 ASYNC SCHEDULING: overlap step N+1 host-prep with step N GPU exec, keep sampled tokens on-device, build next input_ids on-GPU (no per-step host round-trip). CAVEAT: flat scaling also fits within-forward per-layer dependency bubbles (gather→prefill→scatter) — the in-place-via-indices GDN fix is the alternative if the lockstep is within-forward not between-step. Investigation af9c04f running to confirm which + scope the fix.** |