Skip to content

perf(dspark): propose two tokens where the draft already drafts four, and widen NVFP4 activation loads - #896

Closed
coderbench wants to merge 1 commit into
gittensor-ai-lab:mainfrom
coderbench:dspark-nvfp4-loads
Closed

perf(dspark): propose two tokens where the draft already drafts four, and widen NVFP4 activation loads#896
coderbench wants to merge 1 commit into
gittensor-ai-lab:mainfrom
coderbench:dspark-nvfp4-loads

Conversation

@coderbench

@coderbench coderbench commented Aug 20, 2026

Copy link
Copy Markdown

Summary

The headline change: propose two tokens in the narrow band instead of one. The measurement that pinned it at one is stale, and the draft is already paying for the rows it throws away.

Proof of speedup

  • Tested on RTX 5090 (sm_120)

Decode tok/s (dspark-decode@4k, dspark_tau_check, pristine origin/main @ 30ddb88 vs this PR in separate build trees, run alternately on the same box):

decode tok/s
before (main) 67.11
after (this PR, kernel changes only) 77.26

Guards: ctest 19/19 · LOSSLESS=1 verified across repeats · tau unchanged · AR improved (91.77 -> 92.94, floor bar 0.98x).

# BEFORE -- origin/main @ 30ddb88
METRIC AR_TPS 91.7484
METRIC DSPARK_TPS 67.1087
METRIC MEAN_ACCEPT 1.0000
METRIC LOSSLESS 1

# AFTER -- this PR
METRIC AR_TPS 92.9399
METRIC DSPARK_TPS 77.2557
METRIC MEAN_ACCEPT 1.0000
METRIC LOSSLESS 1
METRIC LOSSLESS_RUNS 2

# batched verify forced on (eval-box path), mean of 2 alternating pairs
main     batched 12.776 ms/call  n=128
this PR  batched 12.385 ms/call  n=128

@coderbench coderbench changed the title perf(dspark): widen NVFP4 activation loads and resolve the engage EMA below a whole token perf(dspark): propose two tokens where the draft already drafts four, and widen NVFP4 activation loads Aug 20, 2026
@skyrocket2026

Copy link
Copy Markdown
Member

Closed — the reported baseline does not reconcile with this repo's

Not a judgement on the code. The problem is the measurement, and it is specific.

You report benchmarking against pristine origin/main @ 30ddb88. The eval box measures that same
main at 112.34 tok/s, tau 1.662 on dspark-decode@4k. Your report has main at 67.11 tok/s,
MEAN_ACCEPT 1.0000
.

MEAN_ACCEPT 1.0000 is the load-bearing detail. It means the draft accepted nothing — every
step kept only the bonus token and zero proposals — in both your before and your after run. That is
the degenerate regime where DSpark is just AR plus draft overhead, which is exactly why both your
numbers land below your own AR column (67.11 and 77.26 against 91.75 and 92.94), while main on this
box runs at 1.24x AR.

That matters for this PR specifically, because the headline change is "propose two tokens where the
draft already drafts four."
With the accept count pinned at 1.0, no proposal is ever accepted, so a
change to how many tokens are proposed cannot be what moved your number. Something else in the setup
did, and until the baseline reconciles there is no way to tell what.

Worth checking on your side:

  • Context. The batched verify only arms at or above kEngageMinSeq (1024). Below it the token
    loop runs one target forward per kept token — the same count AR runs — so speculation cannot pay
    there by construction, and tau collapses toward 1.0.
  • Prompt. tau is a function of how predictable the generated text is. Synthetic or arbitrary
    token ids drive the target into degenerate output and produce tau readings that tell you nothing
    about real workloads. The bot scores bench/scripts/bench_prompt_4k.txt tokenized with the
    checkpoint's own tokenizer.json.
  • Checkpoint pair. The scored config is the ModelOpt NVFP4 target plus the released DSpark
    draft. A different target or draft will not reproduce these numbers.

If you can produce a before/after where the main baseline lands near 112 tok/s at tau ~1.66, reopen
and it will be evaluated on its merits — the underlying idea is not the issue here. Note that main
moved several times today (#890, #891, #893, #894), so rebase before re-measuring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants