Bound DS4 W2 restaging and tier execution - #5
Open
OmarB97 wants to merge 1 commit into
Open
Conversation
Author
|
Split series replacing closed monolithic PR #4:
PRs #5-#8 are independently reviewable against main. Their union plus #9 is byte-identical to the frozen validated tree GitHub currently reports the fork-triggered Actions suites on #5 and #9 as |
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…8; bug kacper-daftcode#3 verified below it max_tokens=1 (forward pass only), temperature 0, seed 0, sequential single requests, same process, DETERMINISTIC_MOE=1, prefix caching off: prompt 2048 1 distinct logprob set / 6 repeats 0.000e+00 prompt 2064 6 distinct logprob sets / 6 repeats 4.432e+00 nats A step, not a gradual onset. Bit-identical at or below 2048, every repeat differs above it, out to 24k. NOT chunked prefill. BATCHED_TOKENS=1024 puts the chunking boundary at exactly 2048 too (2x1024), so the two explanations coincided. Re-ran the probe at BATCHED_TOKENS=4096, where everything through prompt 4096 is one chunk: the crossing does not move. The boundary is a property of the sequence -- index_topk=512 x ratio-4 = 2048 compressed candidates, the same boundary as bugs kacper-daftcode#1, kacper-daftcode#2 and the termination collapse. Why it was missed: bug kacper-daftcode#3 verified determinism on a 77-token prompt. Below 2048 the indexer top-k is a no-op, so residual ULP-scale nondeterminism is absorbed and cannot reach the output. The verification ran entirely inside the regime where the bug is structurally invisible -- the same shape as bug kacper-daftcode#2's self-test and vLLM's no-op `high` branch. Third instance of a green test that could not fail. Does not invalidate bug kacper-daftcode#3's fix: below 2048 it is what makes the run bit-identical. But the 5.3% it costs does not buy reproducible greedy decoding at any realistic context length, so that trade wants re-deciding. Whether the residual source also exists below 2048 (masked) or only above it is unknown from output alone. VLLM_DSV4_TOPK_ORACLE with _DET=1 separates the two: logits reproducible but selection differing means order-dependent tie-breaking; logits differing means an upstream race the selection amplifies. Also: --enable-prompt-tokens-details is now always passed. Without it usage.prompt_tokens_details is null and cached_tokens is invisible, so a prefix cache hit cannot be told from a miss at the API. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
… not tie-breaking
Ran the top-k oracle probe (PREFILL=1, ENFORCE_EAGER, no cudagraphs, caching
off) on two byte-identical requests and grouped records by (layer, seq_len).
Not tie-breaking: 20/21 layers had the LOGITS differ between runs, 0/21 had
identical logits with a differing selection, 0/21 had any tie at the cut. Both
runs scored exact:true miss:0 hit:512 -- the selector is exact on inputs that
already differ.
Not masked below the boundary either, which kills the working hypothesis from
the previous commit. With MIN=0:
compressed seq_len 256 0/21 differ
379 0/21
512 0/21 <-- exactly k_select
515 20/21 4.45% <-- first diverging layer 4
Bit-identical at 512 candidates, nondeterministic at 515. The >k_select regime
creates the nondeterminism rather than revealing it.
Exonerates the MoE: det-moe=1 was on and the MoE runs at every layer regardless
of sequence length, so a racy MoE would diverge at 256/379/512 too.
Narrows the suspect: layer 2's logits are identical and its selection is exact
with no tie, so it picks the same 512 indices both runs -- yet layer 4's logits
differ. The perturbation is injected between them by something that engages only
when the selection is real, i.e. the sparse-MLA attention over a scattered
512-of-N gather. Same file family as bug kacper-daftcode#2, fork code.
Divergence compounds with depth, 0.50% at layer 4 to 3.86% at layer 42. Onset
reads as "present by layer 4" not "originates at": finite_max is a scalar
summary and only every other layer carries an indexer.
Next: VLLM_DSV4_DOUBLE_MODULE on the sparse attention module at ctx 2047 is a
direct yes/no on a racy kernel with no inference from downstream logits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…ive; the kernel is not racy
Forward trace (1223 modules, eager, no graphs, caching off), two identical ctx
2047 requests, split on embed_tokens and compared chunk-for-chunk:
chunk 0 1024 tok 256 compressed cand bit-identical
chunk 1 1024 tok 512 bit-identical
chunk 2 15 tok 515 573 modules differ
[36] out same ColumnParallelLinear layers.2.attn.wq_b
[37] out DIFF RowParallelLinear layers.2.attn.wo_b <- input already differs
Layers 0-1, both compressors, indexer_op, indexer and wq_b are all bit-identical.
The injection is between wq_b and wo_b: the sparse-MLA attention core, which is
not an nn.Module and carries no hook. MoERunner's "input same, output differs"
is an artifact of fingerprinting only the first tensor arg -- DeepseekV4MoE
right after it shows IN DIFF, so the MoE inherits, matching the oracle.
The kernel is NOT racy. Standalone, no server, 5 repeats, both the plain SWA
geometry and the engine's dual-cache geometry (packed fp8 compressed cache +
extra_sparse_indices + sinks, T=15), candidate counts 128..1024 including
511/512/513/516: one distinct output everywhere, zero delta. Checked with inputs
reallocated between calls too, which would expose a read of unowned memory (the
bug kacper-daftcode#2 shape), and a poisoned `out` buffer comes back fully overwritten.
It IS order-sensitive. Same index set, permuted per row:
n_extra 256 512 516 1024
max |d| 1.56e-2 1.56e-2 1.56e-2 7.81e-3
~1 bf16 ULP. Candidates accumulate in index order and float addition is not
associative -- bug kacper-daftcode#4's defect family, in the attention instead of Marlin.
Mechanism: below 512 the top-k is a no-op, indices come out in natural order,
order is stable, run reproducible. Above 512 a real selection runs; a varying
order rounds differently and ~1 ULP at layer 2 compounds to 3.86% by layer 42
and 1.6-4.4 nats at the output. Ruled out: kernel race, tie-breaking (0 ties at
the cut), MoE (twice), chunked prefill (boundary does not move with
BATCHED_TOKENS), masking below the boundary (logits identical at 512).
Unproven remaining link: that the selector emits a varying ORDER for the same
set. Inferred by elimination. Confirm by dumping raw topk_indices for two
identical requests and diffing element-wise, not as a set -- the oracle's
exact:true only checked set membership, which is precisely why this hid. If
confirmed the fix is to sort selected indices before attention consumes them,
the same remedy shape as bug kacper-daftcode#3's moe_align canonicalisation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…n is bit-identical
Added VLLM_DSV4_TOPK_ORACLE_RAW=1, which dumps the selected index vector in
emission order. The scored `sel` field could never answer this: exact:true is
computed against torch.topk and reports hit/miss counts, so two runs selecting
the same 512 candidates in a different ORDER both score exact and look the same.
Diffed element-wise across two identical ctx-2047 requests:
layer 2 logits same selection IDENTICAL 0/512 positions differ
layer 4 logits DIFF different set 497
layer 6 logits DIFF same set, reordered 388
... 10 reordered / 10 different-set
Layer 2 is the first indexer layer and the one where the forward trace puts the
injection. There the indexer is completely deterministic -- identical logits AND
an identical index vector, same set and same order -- and the attention still
diverges. So ordering is not the trigger. The reordering and set changes from
layer 4 down are consequences of already-differing logits, not causes.
Kernel cleared at engine page size too: pbs=256 (BLOCK_SIZE) as well as 64, at
n_extra 512/515/516, one distinct output over 5 repeats, poison probe clean.
That leaves the corner: injection in layers.2.attn between wq_b and wo_b, with
identical q, identical selection indices, and a kernel deterministic on
identical inputs. The one unexamined input is the KV CACHE CONTENTS -- mutated
in place, invisible to a module-output fingerprint, which is the same blind spot
that hid bug kacper-daftcode#2. Next: fingerprint the compressed and SWA cache tensors right
before layer 2's read. Also check whether a separate main/SWA selection exists
that sel_idx does not cover (the dump is from the top_k_per_row_prefill call
site only).
Recorded as a latent hazard rather than the cause: the sparse-MLA attention IS
order-sensitive, ~1 bf16 ULP under permutation of an equal index set. Not what
is biting now, but canonicalising indices is worth doing defensively.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…terministic ORDER (upstream)
Raised VLLM_DSV4_TOPK_ORACLE_PREFILL from 1 to 16 -- all query rows of the final
chunk instead of only the last -- and diffed sel_idx element-wise at layer 2:
row 3 logits 3.520913/3.520913 same SAME SET, REORDERED 315/512
row 5 logits 2.665767/2.665767 same SAME SET, REORDERED 384/512
row 10 logits 2.995578/2.995578 same SAME SET, REORDERED 127/512
totals: 10 reordered, 2 identical, 0 different-set; logits same on ALL rows
Identical logits in, identical candidate SET out, different ORDER. The earlier
"layer 2 is clean" reading was an artifact of _PREFILL=1 sampling one row of
fifteen, and it landed on one of the two stable rows.
Chain, every link measured:
1 selector permutes on bitwise-identical logits (above)
2 compute_global_topk_indices_and_lens maps positionally, so the permutation
passes through into extra_sparse_indices
3 gathered KV sequence differs while the set does not:
kv_extra_rows content(set)=same sequence(ord)=DIFF 7680/7680
4 sparse-MLA attention is order-sensitive, ~1 bf16 ULP (1.56e-02)
5 compounds 0.50% at layer 4 -> 3.86% at layer 42 -> 1.6-4.4 nats
Why exactly 2048: below k_select=512 the top-k is a no-op, the list comes out in
natural order, order is stable, run bit-reproducible. Above it a real selection
runs.
UPSTREAM, not the sm89 port. top_k_per_row_prefill is a compiled op from the
official wheel (torch.ops._C, _custom_ops.py:2723), in none of our 73 patches,
and the pristine v0.25.1 baseline calls it identically. The fork's Triton kernel
is exonerated -- deterministic on identical inputs at every geometry tested.
Also fixes the IO probe that hid step 3: its gather fingerprint used
torch.unique (sorted), so it could only see content-as-a-set, never sequence --
the exact property under investigation. Now emits both set_* and ord_*, the
ordered form computed in bounded chunks after the first version OOMed the engine
allocating 1.71 GiB of int64.
Fix: canonicalise (sort) each row's selected indices after the top-k. 512 ints
per row. Same remedy shape as bug kacper-daftcode#3's moe_align, same reason: order-sensitive
arithmetic behind a producer that guarantees no order. Worth upstreaming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…SV4_DETERMINISTIC_TOPK)
_canonicalise_topk() sorts each row's selected indices after the selector runs,
at both call sites: after top_k_per_row_prefill, and where the three decode
selectors converge. Padding is -1 and consumers expect it last, so ascending
sorts against a max sentinel instead of sorting -1 to the front. =2 is the
descending control, mirroring the MoE knob.
Verified with cudagraphs on, realistic config, 6 repeats, max_tokens=1:
ctx 2047 2063 2200 3072 4096 8192 10240 before: 6 distinct / 6
after: BIT-IDENTICAL
ctx 12288 4 distinct 0.783 nats
ctx 16384 6 distinct 1.722 nats
ctx 24000 6 distinct 2.442 nats
Reproducible range goes from "nothing above 2048" to 2048-10240. 5x.
Defaults to 0, unlike DETERMINISTIC_MOE, because the cost is unmeasured
end-to-end. The sort is launch-latency bound -- 0.132 ms/call at [1024,512],
[15,512] and [4096,512] alike, so three launches plus a copy rather than real
work -- which is ~5.7 ms per forward across 43 layers. Brutal against a ~20 ms
decode TPOT if it survives cudagraph capture, near-free if it does not.
Benchmark before flipping the default; fusing where/sort/where is the obvious
mitigation.
A SECOND source remains above ~11k, and it is a different shape: the boundary
sits between 10240 (clean) and 12288 (4 distinct of 6), and it is probabilistic
rather than the clean step 2048 was -- 4/6, then 3/6 at 14336, then 6/6 at
15360. Suspect, unverified: compress_ratios alternates 4 and 128 and the two
take different paths; ratio-128 uses attn_metadata.c128a_prefill_topk_indices,
built in build_c128a_topk_metadata and untouched by this fix. Its
max_compressed_tokens=8192 default is the right magnitude but the wrong
arithmetic (L/128 = 96 candidates at 12288), so do not assume it without
measuring.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…ic + deep-swe Puts the current state at the top of the TP8 runbook so a cold reader does not have to reconstruct it from 1400 lines of investigation: nothing is serving, the repo is clean at 287e206, and the next action is the agentic boot command with prefix caching on. Summarises what settled this session (bug kacper-daftcode#5 root cause + partial fix defaulted off, prefix-caching gate passed, fp32 dot null, prompt-token-details, chat_utils port) and what is deliberately left open (benchmark the TOPK knob, the ~11k source, pc_needle.py never run). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PWdhL5uHMtRNKPK6VKBteH
the-crypt-keeper
pushed a commit
to the-crypt-keeper/vLLM-sm89
that referenced
this pull request
Aug 4, 2026
…er-daftcode#5 Benchmarks: full backend x speculation matrix at c1/c4/c8 (Triton and FlashInfer, no-spec / DSpark k=3 / MTP k=1 / MTP k=2), the k-selection table with acceptance rates, the synergy result, and the c4-c8 crossover. MTP k=1 measured for the first time on this benchmark and it BEATS k=2 everywhere (99.84/204.12/290.37 vs 86.80/170.49/221.78) -- k=2 drafts twice as much for near-identical acceptance length (1.31 vs 1.26) at 15.6% vs 25.8% acceptance. Earlier notes recommending k=2 were measured on a different harness. Characters of the two spec methods differ usefully: DSpark k=3 has the higher peak (+81% single-stream) but pays 15% at c8; MTP k=1 gives +52% single-stream and lands within 2% of no-spec at c8, so it is the safer default for a server covering both interactive and batch traffic. Quickstart: install sections for the FlashInfer sm89 wheel (with the three gotchas -- version-check bypass, jit-cache shadowing, ninja/nvcc on PATH) and for extracting the MTP draft head from the preview checkpoint. Knobs: DETERMINISTIC_TOPK, SPARSE_MLA_FORCE_TRITON/FLASHINFER, plus a launcher-level table for BACKEND/SPEC/SPEC_TOKENS/SPEC_MODEL/CUDAGRAPH_SIZES. What was broken: four -> five, adding the reasoning_content response alias with the deep-swe numbers (PASS 2/15 -> 7/15, matching cloud) and the token-profile convergence that shows the mechanism. Lineage: adds yhfgyyf/vllm-deepseek-v4-sm89 (independent Ada lineage, solved the kernel gap by implementing the sm89 path inside FlashInfer's SM120 kernel) and guqiong96/Lvllmds4-x (lk_moe CPU-GPU hybrid MoE, for boxes short on VRAM).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cold DS4 W2 restaging could hard-wedge a 5090 host, while live tier-manager passes could overlap replay/pin lifecycles and destabilize quality. The implementation needs bounded staging, fail-closed headroom checks, and explicit forward/tier ownership before the lane can be release-safe.
What changed
How to review
Review
patch/vllm-moet-v0.24.0.patchas the generated source artifact, usingpatch/FILES.txtas its lost-line guard. Focus on the store/loader path, forward exclusion lifecycle, and split/base coupling; then inspect the two direct GPU fixtures.Evidence
This exact patch is SHA-256
241ba984b1c56f5dc7adbc8d7f519d60b5746024bf7dfeb875e3546a668e79a7and was baked into imagesha256:fc6e1244d60855fe45ccc0236daaaa722abcb8d354200eb92aca104bd954d3f2. All 70 baked source files matched a fresh apply. The exact image passed 21 focused CUDA tests, 37 store-safety tests, 10 MXFP4 stream tests, the GPU backend matrix, and both E=8 forward fixtures.Verification
python3 tools/check_patch_files.pypython3 -m py_compile tools/test_moe_w2_forward.py tools/test_three_tier_split.pygit diff --checkRisks / gaps
The generated canonical patch is the irreducible review object in this stack (
+3,947/-763by itself). Splitting or regenerating it would change the image identity and require a new release build. Runtime throughput was not benchmarked; this slice establishes implementation safety and correctness.Collaborators