perf(dspark): propose one token, not three, once the draft is carrying real context (+9.4% dspark-decode@4k) - #878
Conversation
…g real context (+9.4% dspark-decode@4k, tau unchanged)
|
Multi-rep losslessness at the scored context, as promised in the description — same build, ctx=4096, production split counts: Every repeat re-runs the whole generation including the token-loop prompt prefill, and the mean accept lands on 1.0756 again — the same value main measures, which is the point: this PR removes proposals acceptance was not using rather than trading acceptance for throughput. |
skyrocket2026
added
eval-dspark:S
sparkinfer DSpark auto-eval —
|
| metric | value |
|---|---|
| label | eval-dspark:S |
| scored at | DSpark speculative decode @ ctx=4k on the ModelOpt NVFP4 checkpoint |
| PR DSpark tok/s | 45.36 |
| main DSpark tok/s | 43.06 |
| speedup vs main | 1.053× (+5.3%) |
| PR AR tok/s (floor) | 47.40 |
| main AR tok/s (floor) | 47.39 |
| AR vs main (floor) | +0.0% |
| DSpark vs AR | 0.957× — above 1.0 means speculation finally pays |
| mean accept τ | 1.085 (main 1.085, ceiling 7) |
| accuracy gate | ✅ top1=1.000 (bar >=0.9) · KL=0.0000 (bar <=0.1) |
| losslessness gate | ✅ DSpark matches the AR reference token-for-token, verified across 3 independent runs |
| mean accept τ floor | ✅ 1.0847 vs main 1.0847 (bar ≥95%) |
| qwen3.8 (ModelOpt) guard @16k | ✅ no regression (decode+prefill) — decode 89.5 tok/s · prefill 13555 pp |
| qwen3.6 guard @16k | ✅ no regression (decode+prefill) — decode 474.5 tok/s · prefill 28368 pp |
| PPL PR / main | 3.204 / 3.204 |
| Polaris receipt | collected, not signed (no key configured) |
| commit | b545b4f2a |
ok
Scored on the pinned eval box vs same-box origin/main: DSpark speculative decode throughput at ctx=4k on the ModelOpt NVFP4 checkpoint, with the AR reference measured in the same process and the same model load. Both a regression in AR decode and any divergence from the AR token sequence are hard REJECTs — a speculative decoder that is fast because it skips verification is not faster, it is wrong. τ is the lever, and the row above reports it against a block_size of 7. This is informational, not a judgment on your PR: a none label just means no measurable DSpark decode@4k speedup was verified, which is expected and fine if that isn't what your change is about. Automated — merge behaviour depends on SPARKINFER_DSPARK_AUTOMERGE.
skyrocket2026
added
dspark-needs-rebase
Closed: sparkinfer DSpark auto-eval —
|
…ref's own checkout The harness lives in the repo, so each ref was being measured with its own copy of it. That is harmless while the harness is stable and catastrophic the moment it changes: every PR branched before the change gets compared against a baseline measured by a different instrument. It cost us the best PR of the day within an hour of the pin removal. #878 was branched before f3b2e04, so its checkout still carried dspark_tau_check WITH the NSPLITS=1 pin. It measured dspark 45.37 / ar 47.40 -- the pinned regime -- against a main baseline of 74.07 / 90.16 measured without the pin. The bot read that as -47.5% on ar-decode@4k, labelled it REJECT and auto-closed it. Nothing was wrong with the PR: it was the one change all day that closed the DSpark/AR gap (0.909 -> 0.957) with AR flat and tau held. Reopened by hand. Now both refs take runtime/examples/dspark_tau_check.cpp and bench/scripts/bench_prompt_4k.txt from origin/main after checkout, so a round always compares two builds through one ruler. Same pattern the bot already uses for eval/polaris, and consistent with HARNESS_PATHS -- a PR may not change these files anyway, so sourcing them from main costs a contributor nothing. Fails the run loudly if the pin cannot be applied, rather than silently measuring with whatever the ref happened to carry.
skyrocket2026
removed
eval-dspark:REJECT
|
Reopened — the REJECT was our bug, not yours. Apologies for the noise. An hour before that round, we removed a The harness lives in the repo, and the bot was measuring each ref with its own copy of it. Your branch predates the removal, so it was measured with the pin while Two builds, two different rulers. Nothing about your change caused it. For the record, your PR was the best result of the day. Under the v1 numbers it scored S at +5.3%, but the tier undersold it: it was the only change that moved DSpark toward break-even — ratio 0.909 → 0.957, with AR flat and τ held at 1.085. The three attention PRs that outscored it (L, XL, L) each lifted AR more than the speculative path, so they widened the gap; and measured at production split counts, all three delivered ~0% (production decode at 4k went 93.69 → 93.64 across the day). Yours was the one that made speculation genuinely cheaper. Two things have been fixed:
Your reasoning about the depth-3 choice resting on pin-inflated measurements was also correct, and it is part of why the pin was removed. Nothing is needed from you — it will be re-scored automatically on the next round. |
skyrocket2026
added
eval-dspark:M
sparkinfer DSpark auto-eval —
|
| metric | value |
|---|---|
| label | eval-dspark:M |
| scored at | DSpark speculative decode @ ctx=4k on the ModelOpt NVFP4 checkpoint |
| PR DSpark tok/s | 82.12 |
| main DSpark tok/s | 74.82 |
| speedup vs main | 1.097× (+9.7%) |
| PR AR tok/s (floor) | 90.07 |
| main AR tok/s (floor) | 90.23 |
| AR vs main (floor) | -0.1% |
| DSpark vs AR | 0.912× — above 1.0 means speculation finally pays |
| mean accept τ | 1.076 (main 1.076, ceiling 7) |
| accuracy gate | ✅ top1=1.000 (bar >=0.9) · KL=0.0000 (bar <=0.1) |
| losslessness gate | ✅ DSpark matches the AR reference token-for-token, verified across 3 independent runs |
| mean accept τ floor | ✅ 1.0756 vs main 1.0756 (bar ≥95%) |
| qwen3.8 (ModelOpt) guard @16k | ✅ no regression (decode+prefill) — decode 89.0 tok/s · prefill 12706 pp |
| qwen3.6 guard @16k | ✅ no regression (decode+prefill) — decode 472.7 tok/s · prefill 28365 pp |
| PPL PR / main | 3.204 / 3.204 |
| Polaris receipt | collected, not signed (no key configured) |
| commit | b545b4f2a |
ok
Scored on the pinned eval box vs same-box origin/main: DSpark speculative decode throughput at ctx=4k on the ModelOpt NVFP4 checkpoint, with the AR reference measured in the same process and the same model load. Both a regression in AR decode and any divergence from the AR token sequence are hard REJECTs — a speculative decoder that is fast because it skips verification is not faster, it is wrong. τ is the lever, and the row above reports it against a block_size of 7. This is informational, not a judgment on your PR: a none label just means no measurable DSpark decode@4k speedup was verified, which is expected and fine if that isn't what your change is about. Automated — merge behaviour depends on SPARKINFER_DSPARK_AUTOMERGE.
|
Auto-merged as the round's |
…s (+25% at ctx=4k) The narrow band existed because a 4-wide verify could not pay for itself. #878 measured depth 1 beating depth 3 by 9.4%, correctly, when a 4-row batched verify cost 2.62 target forwards against a mean accept of 1.92 -- C > tau, so every extra proposal lost. The kernel work in this branch took that to 1.85, C < tau, and the sign of the trade flips. Re-measured at ctx=4096, same prompt, same build, LOSSLESS in every case: depth 1 (2-wide) tau 1.3611 72.28 tok/s 0.798x AR <- what #878 chose, still the default depth 3 (4-wide) tau 1.9231 90.57 tok/s 1.010x AR <- and what now wins depth 6 (7-wide) tau 1.9231 53.81 tok/s 0.600x AR tau is identical at 4-wide and 7-wide -- acceptance saturates at ~1.92 -- so 4 is the widest block that buys anything and 7 only buys cost. Without this the whole branch is invisible: every cost improvement lands on a batched verify the default configuration barely uses, and the scored dimension keeps reading 0.798x. The band narrows to what it was really describing -- the range where the batched verify CANNOT arm. Below kEngageMinSeq the token loop runs one target forward per kept token, exactly what AR runs, so proposing more is pure draft overhead and depth 1 stays. At or above it the batched pass arms and depth 3 is the operating point. The >= kDeepMinSeq branch keeps its 7, unmeasured here and measured separately.
…rify arms" This reverts d37034c. The measurement behind it was invalid. It was taken on a synthetic prompt -- token ids 1000..5095, i.e. gibberish -- which drives the target into degenerate repetition (at ctx=128 the AR output is literally '197 320 197 320 ...'). Repetition is trivially predictable, so tau inflates to 1.9231 and the 4-wide block looks like it pays. It does not. Re-measured at ctx=4096 on the corpus the eval bot actually scores (bench/scripts/bench_prompt_4k.txt, tokenized with the checkpoint's own tokenizer): prompt tau DSpark vs AR prose (bot's corpus) 1.4118 66.57 0.736x code (C++) 1.1163 68.13 0.759x eval_corpus 1.2973 69.24 0.772x synthetic ids 1.9231 90.66 1.011x <- what d37034c measured and on that corpus the depth sweep runs the other way: depth 1 (2-wide) tau 1.2632 78.80 tok/s 0.877x AR <- #878's choice, correct depth 3 (4-wide) tau 1.4118 66.23 tok/s 0.739x AR depth 6 (7-wide) tau 1.4545 40.44 tok/s 0.451x AR #878's narrow band was right and stays. The verify-cost work in this branch is unaffected -- it is kernel-level and prompt-independent, and C at N=2 still improved 1.49 -> 1.23 forwards -- but it is not enough to make a wider block pay at the acceptance this model actually achieves on real text. Lesson worth keeping: never measure tau on synthetic token ids. Speculative-decoding throughput is a function of how predictable the generated text is, so a degenerate continuation measures the harness, not the decoder.
2 participants
Summary
Below the deep threshold the draft proposes 3 tokens. At this model's acceptance those extra
proposals are never accepted, so they are pure draft cost — and the cost is not constant: each
proposal is another row of the draft's own attention over the same KV, so it grows with context
while what it buys stays at zero.
The measurements the depth-3 choice rests on were taken with the harness's
NSPLITS=1pin inplace. At the split counts the server actually runs, the attention that pin inflates is cheap, the
draft's fixed cost is a much larger share of the step, and the balance moves. Measured on RTX 5090
at production split counts, one binary,
LOSSLESS 1on every run:Acceptance is identical at 4k and 2048 — depth 3 buys exactly nothing there. 128 is the one
context that prefers the wide block, and its acceptance column says why: at depth 3 the draft
accepts essentially nothing (1.0159), the idle-draft rule stops drafting, and the stream runs at
AR speed. Narrowing raises acceptance just enough (1.1034) to keep the draft alive without being
worth its cost. So this does not narrow always — it narrows once the draft is carrying a context
whose per-row cost the deeper proposals cannot repay, which is what the 512 tie and everything
above it show. Below
kNarrowMinSeq(512) nothing changes, and the>= kDeepMinSeqbranch keepsits 7 — long context is where acceptance actually climbs and it is measured separately.
Decided once per generation, not per step, and that distinction is load-bearing. The verify
graph
dflash_warm_verifycaptures is sized from this depth; narrowing per step under a warmgraph built for 4 rows recovers only 0.7% of the 9.4%, because the mismatched graph costs more
than the rows it saves. Both were measured; the static form is the one that pays.
Proof of speedup
sm_120)Decode tok/s (
dspark_tau_check <modelopt> <dspark> 128 <4096-token prompt>,METRIC DSPARK_TPS,production split counts):
+9.4% dspark-decode@4k. Main and PR binaries interleaved round-robin:
No context regresses, and the two below the band are unchanged by construction:
Correctness
LOSSLESS 1on every run above, including the interleaved rounds and all four guard contexts.Mean accept is unchanged: 1.0756 on main, 1.0756 with this PR, at 4k — this PR does not buy
throughput with acceptance, it removes draft rows that acceptance was never using. That is the
tau floor's own question, answered by identity rather than by tolerance.
Multi-rep losslessness at the scored context is in a comment below (running as this was written;
SPARKINFER_DSPARK_SPEC_REPSat ctx=4096).SPARKINFER_DFLASH_NARROW_MINSEQpins the band boundary (default 512) andSPARKINFER_DFLASH_PROPOSALSstill overrides the depth outright, so both arms come out of onebinary.